MineMeld Articles

Featured Article
Use MineMeld DAGPusher to create a basic NAC application
View full article
xhoms 3 weeks ago
7,267 Views
1 Reply
1 Like
Overview In some cases you might be asked to enable the YouTube application but only for a few set of videos or the ones in a given playlist. MineMeld includes an experimental miner prototype that can extract the video items in a YouTube playlist and convert them into a URL list that can be imported into your Internet Gateway Palo Alto Networks Firewall to achieve such a goal.   Theory of operations There are three components that are needed to implement this use case: A running MineMeld instance A google account and a YouTube Data API v3 key (can be obtained for free from console.cloud.google.com) An Internet Gateway PANOS device The YouTube miner will use the provided API Key and the PlayList unique ID to grab the list of videos. They will be converted in a set of URL's (https://www.youtube.com/watch?v=...) that can be aggregated and placed into an output feed. If you need additional videos then you can add a stdlib.listUrl miner node and manage your own exceptions.   The security policy in the FW will need two rules: A bottom rule enabling google-base and youtube-base with a URL filtering profile blocking the url "www.youtube.com/watch". This way, although the streaming of videos is enabled, the YouTube application logic is broken at the time the user requests a given playback (/watch?v=...) An upper rule overriding the URL filter for the list of videos provided by the MineMeld feed A ssl decryption rule is needed to allow the FW full access to the URL details to successfully apply the filters.   Step 1. Deploy MineMeld First, visit https://live.paloaltonetworks.com/t5/MineMeld/ct-p/MineMeld and select the article (from the top right) about installing and running MineMeld appropriate to your environment. Note, if using the VMWare desktop instructions (https://live.paloaltonetworks.com/t5/MineMeld-Articles/Running-MineMeld-on-VMWare-desktop/ta-p/72038) you can go ahead with the "Super fast setup" but please download the cloud-init ISO and mount it on first boot. Assuming an IP comes via DHCP and you have internet access, your VM will automatically be updated  to the latest version of Minemeld.   Make note of MineMelds IP address (from an ifconfig) and login from your browser (defaults to username: admin / password: minemeld)   Step 2. Get your YouTube Data API v3 key Connect to https://console.cloud.google.com and login with your Google Account. If this is the first time you access the cloud console then an empty dashboard will be shown.     Navigate to the API Manager -> Library and then click on YouTube Data API.       You can use any existing project to attach the YouTube API also. If this is your first time or if you do not have any project then you must create a new one. Each Google Account is provided with a quota of up to 12 projects for free.       After clicking the "Create" button you will be send back to the YouTube Data API screen where you must click on "Manage" to allow this project access YouTube data. Once enabled the "Create credential" button will be shown for you to move on.         Fill the form as shown in the following screen capture and click on "What credentials do I need?" to disclose the API key.       This is the API Key you'll be asked for in the step 4. So copy it to a safe place.    Step 3. Get the ID of the YouTube playlist you want to mine. This is the easiest part. We'll cover three common demands: Videos from a known PlayList Videos on a specific YouTube channel Videos upload by a known User In the case you already know, just navigate YouTube until you reach the playlist you want to enable and copy its ID from the URL. The following screen captures are for the Ligthboard Series playlist in the Palo Alto Networks channel.            If you need to get the videos from a specific channel, then you can take advantage of an "under-the-hood" mapping in YouTube between the Channel ID and its corresponding Upload Playlist ID. Let's take, as an example, the Live Community Channel in Palo Alto Networks' YouTube Account.     This channel's ID is UCPRouchFt58TZnjoI65aelA so its corresponding upload playlist id (containing the 87 videos uploaded to this channel) will be UUPRouchFt58TZnjoI65aelA. In other words: you just need to change the second character "C" with a "U".   Something equivalent happens for YouTube users. Each user in YouTube has an internal Channel ID that, if transformed, becomes the user's upload PlayList ID. To display any YouTube user internal Channel ID just play a video from that user and click on his name.     In this case we're using the Palo Alto Networks YouTube user as a example. Click on the user name to navigate to the YouTube user profile page. Note in the URL the Channel ID for this user.     In this case the Channel ID for the Palo Alto Networks user is UC2UPStk47kvhBn8P7Q5BaAg and its corresponding upload playlist ID (containing all videos uploaded by this user) will be UU2UPStk47kvhBn8P7Q5BaAg.   Step 4. Configure your MineMeld instance The experimental YouTube paylist miner is provided as an external feature that can be added to your MineMeld instrance from https://github.com/PaloAltoNetworks/youtube-miner   In this step you'll use the YouTube API Key and PlayList ID to configure the miner and generate the URL feed. First click on "CONFIG" to expose your current configuration. In the bottom right part of the screen you'll locate the icon to access the prototype library. Open it and locate the "youtubeminer.playlistMiner" prototype.     Click on it and create a new prototype from that base     Fill the form using your YouTube API Key and Playlist ID from Steps 2 & 3.     The next step is to add the recently created prototype as an engine node. To achieve this just go back to the prototype library, locate the recently created one and click on "CLONE".         Provide a descriptive name for the node and click on "OK" to attach it to the engine's configuration.     Next step is attach a URL processor to the engine's configuration and to connect its input to the YouTube miner node. In this case we do not need to create a new prototype. Just "CLONE" the stdlib.aggregatorURL, provide a descriptive name for the node, and bind its input to the miner.     And, finally, we must attach an output node. We can just "CLONE" the stdlib.feedHCGreen and bind its input to the aggregator we deployed in the previous step.     Now it is time to commit the configuration and to check that the output node is publishing the expected list.               Step 5. Configure the PAN-OS Internet Gateway device This is the last step and the documentation bellow is for a "green field" deployment and must be taken as guidance to modify your existing policy to provide this use case.   First of all a SSL decryption rule is needed to expose the URL details inside the YouTube application. In this example we're enabling "forward proxy" decryption for all SSL sessions from "trust" to "untrust" for the URL category "streaming-media" which the YouTube application belongs to. If this is your first time with the SSL Decryption feature then look in the PANOS Knowledge Base for articles on how to configure it (Trust CA Certificates, Decryption Profiles, etc.)     You might want, as well, to deny the application named "quic". It maps to the experimental protocol used by Google's Chrome browser when accessing Google services like YouTube. Denying this application will force the browser to fail back to TLS and avoid Chrome user bypassing the decryption policy.   Next we need a "custom URL category" that targets the "www.youtube.com/watch" URL and a URL Filtering Profile that blocks it.         We will use the URL Filtering profile in the first rule that enables "google-base" and also in the first rule that enables "youtube-base". If you do not have such a rules then just create a new one as shown in the following screen capture.     At this moment in time you might want to check that the YouTube application logic has been broken. Go to www.youtube.com. You should be able to navigate the application but a URL Filtering Block page will be shown as soon as you attempt to playback any given video.     Now we have to configure the PAN-OS device with an External Dynamic List connected to the MineMeld output feed created in the step 4.     And the corresponding URL Category must be used in a new security rule (above the previous one) to override the URL Filtering Profile for the videos in the playlist we want to enable. Note that we're using this URL Category as a matching criteria in the rule and not inside a URL Filtering Profile.      With this new configuration only the videos in the mined playlist should be enabled.          
View full article
xhoms 3 weeks ago
39,275 Views
25 Replies
4 Likes
Introduction Although MineMeld was conceived as a threat sharing platform, reality has shown many users are taking advantage of its open and flexible engine to extract dynamic data (not threat indicators) from generic APIs. The highly successful case of extracting O365 dynamic data (IP's, domains and URL's) from its public facing API Endpoint Many users relying on MineMeld to track the public IP space from cloud and CDN providers like AWS, Azure, CloudFlare, Fastly as a much more robust and scalable alternative to mapping them with FQDN objects. Or even people using MineMeld to extract the list of URL's to videos published in specific YouTube playlist or channels via the corresponding Google API. All these are examples of MineMeld being used to extract dynamic data from public API's.   Depending on the source, a new class (python code) may be needed to implement the client-side logic of the API we're willing to mine. But, in many case, the already available ready-to-consume "generic classes" could be used instead. This way the user could "mine" its generic API without the need to deep dive into the GitHub project contribution.   The "generic classes" There are, basically, three "generic classes" that can be reused in many applications: The HTTPFT class: Create a prototype for this class when you need to extract dynamic data from content delivered in HTML or PlainText (text/plain, text/html) The SimpleJSON class (I love this one!): Do you need to extract dynamic data from an API that delivers the response as a JSON Document? You're all set with a protorype of this class! The CSVFT class: Some services still use variants of CSV (delimiter-based multi-field lines) to deliver its content. The following is the rule of thumb that will let you know if the API you want to extract dynamic data from can be "mined" using MineMeld by providing just a prototype for one of these classess (without providing a single line of code!) The transport must be HTTP/HTTPS None or basic authentication (user + password) Single transaction (one call retrieves the whole indicator list – no pagination) Indicators are provided in plain, html, csv or json format. The following sections in this article will teach you how to use these generic classes to mine an example API that provides real-time temperature for four MineMeld-relateed cities in the world:   Format API URL CSV https://test.minemeld.com/csv HTML https://test.minemeld.com/html JSON https://test.minemeld.com/json   Mining a CSV API We will start with CSV because it is, probably, the easiest one between the generic classes. The theory of operations is: The CSVFT class will perform a HTTP/S API Call without (or basic) authentication. The expected result will be table-like document where every line will contain an indicator plus additional attributes separated by a known delimiter. Before the CSV parser kicks in, a regex pattern will be used to discard lines that should not be processed (i.e. comments) The prototype will provide configuration elements to the CSV parser to perform the correct field extraction from each line.   First of all, lets call the demo csv api and analyze the results:   Request -> GET /csv HTTP/1.1 Host: test.minemeld.com Response Headers <- HTTP/2.0 200 OK content-type: text/csv content-disposition: attachment; filename="minemeldtest.csv" content-length: 432 Response Body <- # Real-Time temperature of MineMeld-related cities in the world. url,country,region,city,temperature https://ajuntament.barcelona.cat/turisme/en/,ES,Catalunya,Barcelona,12.24 http://www.turismo.comune.parma.it/en,IT,Emilia-Romagna,Parma,16.03 http://santaclaraca.gov/visitors,US,California,Santa Clara,8.98   The API returns a test/csv content and suggests us to store the results as an attachment with the name minemeldtest.csv. Regarding the content, it looks like 4 data records are provided with up to 5 fields each: url, country, region, city and temperature. There are, as well, two lines that do not provide any value and that should be discarded (the one with the comment and the one withe the field headers) And, as for the CSV parsing task, it looks like the fields are clearly delimited by the comma character. We're ready to go to configure our prototype to mine this API with the CSVFT class.   Step 1: Create a new prototype using any CSVFT-based one as starting point. We will use the prototype named "sslabusech.ipblacklist" as our starting point. Just navigate to the config panel, click on the lower right icon (the one with the three lines) to expose the prototype library and click on the sslabusech one.         Cliking on the sslabuse prototype will reveal its configuration as shown in the following picture.       The most important value in the prototype is the class is applies to. In this case, the CSVFT one we want to leverage. Our mission is to create a new prototype and to change its configuration to accomplish our goal to mine the demo CSV API. The following is the set of changes we will introduce: Name, Description and Tags (to make it searchable in the prototype library) Inside the CONFIG section: We will replace url with https://test.minemeld.com/csv We will change the indicator type to URL and set the confidence level to 100 Provide our own set of fieldnames Define the ignore_regex pattern as "^(?!https)" (to discard all lines except the ones starting with "https") Describe the source_name as minemeld-test Simply click on the NEW button and modify the prototype as shown in the following picture.       Please, take a closer look to the fieldname list and realize the first name in our prototype list to be "indicator" (in the CSV body the first field was suggested to be "url" instead). The CSV engine inside the CSVFT class will extract all comma separated values from each line and use the one matching the column named "indicator" as the value containing the indicator we want to extract. Any other fieldname will be extracted and attached as additional attributes to the indicator.   Step 2: Clone the prototype as a working node (miner) in the MineMeld engine Clicking on OK will store this brand new prototype into the library and the browser will be sent to it. Just change the search field to reveal our csv prototype and then click on it.       Now it is time to clone this prototype into a working node into the MineMeld engine. So just click on the CLONE button, give the new miner node a name and commit the new configuration.         Step 3: Verify the node status. Once the engine restarts you should see a new node in your MineMeld engine with 4 indicators in it. Click on it, then click on its LOG button and, finally, click on any log entry to reveal the indicator details.           As shown in the last picture, the extracted indicators are of URL type and additional attributes like city, region, country and temperature are attached to it.   Other optional configuration parameters supported by the CSVFT class are: fieldname: in case it to be null, then the values extracted from the first parsed lines will be used as fieldnames (remember that one of the fields must be named "indicator") delimiter , doublequote , escapechar , quotechar and skipinitialspace control the CSV parser behavior as described in the Python reference guide   Mining a HTML API In this section you will be provided with steps needed to use the HTTPFT class to mine dynamic data exposed in the response to a HTTP request (typically text/plain or text/html). If you have not done so, please review the complete process described in the section "Mining a CSV API" to understand concepts like "creating a new prototype", "cloning a prototype as a working node", etc.   To build a new HTTPFT class we first need base prototype that already leverages this class. In this example we will use the prototype named dshield.block as the base.       Let's take a deeper look to the HTML API response to figure out how to generate a valid prototype to accomplish our mission.   Request -> GET /html HTTP/1.1 Host: test.minemeld.com Response Headers <- HTTP/2.0 200 OK content-type: text/html content-length: 1626 Response Body <- <!DOCTYPE html><html><head><link rel="stylesheet" href="https://stackpath.bootstrapcdn.com/bootstrap/4.1.1/css/bootstrap.min.css" integrity="sha384-WskhaSGFgHYWDcbwN70/dfYBj47jz9qbsMId/iRN3ewGhXQFZCSftd1LZCfmhktB" crossorigin="anonymous"><title>Real-Time temperature of MineMeld-related cities in the world.</title></head><body><div class="container"><table class="table-striped"> <tr><td><code>city</code></td><td><code>country</code></td><td><code>region</code></td><td><code>temperature</code></td><td><code>url</code></td></tr> <tr><td><code class="small">Barcelona</code></td><td><code class="small">ES</code></td><td><code class="small">Catalunya</code></td><td><code class="small">12.24</code></td><td><code class="small">https://ajuntament.barcelona.cat/turisme/en/</code></td></tr> <tr><td><code class="small">Parma</code></td><td><code class="small">IT</code></td><td><code class="small">Emilia-Romagna</code></td><td><code class="small">16.03</code></td><td><code class="small">http://www.turismo.comune.parma.it/en</code></td></tr> <tr><td><code class="small">Santa Clara</code></td><td><code class="small">US</code></td><td><code class="small">California</code></td><td><code class="small">8.98</code></td><td><code class="small">http://santaclaraca.gov/visitors</code></td></tr> </table></div></body></html>   So, what do we have here? A HTML table whose rows are provided in individual file lines and with each value in its own table cell.   First of all we have to get rid of all lines not belonging to table rows. We can achieve this with the ignore_regex class configuration parameter.   ignore_regex: ^(?!<tr><td>)   Next, we need a regex pattern to extract and transform our values from each line. The HTTPFT class leverages Python's re module and accepts configuration parameters both for the indicator itself and any additional attribute. Any Regular Expression strategy will be valid. We will use the following one in this example:   (<td><code class="small">([^<]+)<\/code><\/td>)(<td><code class="small">([^<]+)<\/code><\/td>)(<td><code class="small">([^<]+)<\/code><\/td>)(<td><code class="small">([^<]+)<\/code><\/td>)(<td><code class="small">([^<]+)<\/code><\/td>)   It is a large expression with 10 capturing groups. The first capturing group (\1) extracts the first cell and the second capturing group (\2) the value inside that given cell. Group 3 extracts cell number 2 and group 4 the value inside that second cell. And so on.   As the indicator (the URL) is in the first cell, then the corresponding configuration to achieve our goal must be:   indicator:     regex: '(<td><code class="small">([^<]+)<\/code><\/td>)(<td><code class="small">([^<]+)<\/code><\/td>)(<td><code class="small">([^<]+)<\/code><\/td>)(<td><code class="small">([^<]+)<\/code><\/td>)(<td><code class="small">([^<]+)<\/code><\/td>)'     transform: \2   For the remaining attributes we can leverage the same regular expression but with different transformations.   fields:     country:         regex: '(<td><code class="small">([^<]+)<\/code><\/td>)(<td><code class="small">([^<]+)<\/code><\/td>)(<td><code class="small">([^<]+)<\/code><\/td>)(<td><code class="small">([^<]+)<\/code><\/td>)(<td><code class="small">([^<]+)<\/code><\/td>)'         transform: \4     region:         regex: '(<td><code class="small">([^<]+)<\/code><\/td>)(<td><code class="small">([^<]+)<\/code><\/td>)(<td><code class="small">([^<]+)<\/code><\/td>)(<td><code class="small">([^<]+)<\/code><\/td>)(<td><code class="small">([^<]+)<\/code><\/td>)'         transform: \6     city:         regex: '(<td><code class="small">([^<]+)<\/code><\/td>)(<td><code class="small">([^<]+)<\/code><\/td>)(<td><code class="small">([^<]+)<\/code><\/td>)(<td><code class="small">([^<]+)<\/code><\/td>)(<td><code class="small">([^<]+)<\/code><\/td>)'         transform: \8     temperature:         regex: '(<td><code class="small">([^<]+)<\/code><\/td>)(<td><code class="small">([^<]+)<\/code><\/td>)(<td><code class="small">([^<]+)<\/code><\/td>)(<td><code class="small">([^<]+)<\/code><\/td>)(<td><code class="small">([^<]+)<\/code><\/td>)'         transform: \10   Combine all these configuration parameters into our desired HTTPFT class prototype as shown in the following picture.       And, finally, clone this prototype as a working (miner) node into the MineMeld engine to verify it is working as expected.        Other optional configuration parameters supported by the HTTPFT class are: verify_cert : It controls if the SSL cetificate should be verified (default - true) or not (false) user_agent : To use a customer provided HTTP User Agent string encoding : To deal with non UTF-8 (default) content.   Mining a JSON API (Please review the complete process described in the section "Mining a CSV API" to understand concepts like "creating a new prototype", "cloning a prototype as a working node", etc.)   The SimpleJSON class features a JMESPath engine to process any JSON document returned by the API call. Take your time to visit http://jmespath.org/ and follow the tutorial to be able to understand all concepts we're going to share in this section.   The following is the theory of operations for JSON miners: The node will perform a non (or basic) authenticated HTTP/S requests and a JSON document is expected to be received in the response. The document will be transformed with a JMESPath expression that MUST convert the original document into an array of simple objects (just key-value pairs). Finally, the indicator key will extract the main value and any additional field key will be extracted and attached as additional attributes to the indicator. Let's take a closer look to the test JSON API we're going to mine:   Request -> GET /json HTTP/1.1 Host: test.minemeld.com Response Headers <- HTTP/2.0 200 OK content-type: application/json content-length: 861 Response Body <- { "description": "Real-Time temperature of MineMeld-related cities in the world.", "result": [ { "url": "https://ajuntament.barcelona.cat/turisme/en/", "country": "ES", "region": "Catalunya", "city": "Barcelona", "temperature": 12.24 }, { "url": "http://www.turismo.comune.parma.it/en", "country": "IT", "region": "Emilia-Romagna", "city": "Parma", "temperature": 16.03 }, { "url": "http://santaclaraca.gov/visitors", "country": "US", "region": "California", "city": "Santa Clara", "temperature": 8.98 } ] }   The JSON document looks quite easy and with a element (result) that already provides us the needed array of objects. So, our JMESPath extractor will be:   extractor: result   You can check the expressión in the JMESPath site to verify this expression will return the following array of objects   [ { "url": "https://weather.yahoo.com/country/state/city-772777/", "country": "Spain", "region": "Catalunya", "city": "Sant Cugat Del Valles", "temperature": "22" }, { "url": "https://weather.yahoo.com/country/state/city-719975/", "country": "Italy", "region": "Emilia-Romagna", "city": "Parma", "temperature": "21" }, { "url": "https://weather.yahoo.com/country/state/city-2488836/", "country": "United States", "region": "California", "city": "Santa Clara", "temperature": "20" } ]   At this point we just need to identify the object attributes that contains 1) the indicator itself and 2) any additional attribute we want to attach to the indicator. In our case, the configuration for it will be:   indicator: url fields: - country - region - city - temperature   It is time to put all these configuration statements into a SimpleJSON class prototype.  We can use, for example, the aws.AMAZON standard library prototype as the base.         Did you noticed the "json" prefix in all extracted additional attributes? You can control that and a few other behaviors of the class with the following optional class configuration elements: prefix: that will be attached to any additional attribute attached to the indicator. verify_cert : It controls if the SSL cetificate should be verified (default - true) or not (false). headers : A list that allows you add additional headers to the HTTP request. client_cert_required , cert_file and key_file can be used to force a HTTP request with client certificate authentication. Bonus Track: using indicator attributes Wondering why would anyone extract additional attributes from the feed and not just the indicator value?   Lets's imagine we want to provide two feeds with Yahoo Weather urls of cities: One of them will be called "time-to-beach" and will list URL's of cities where population is over 30°C and, probably, preparing themselves to go to the beach or outdoor swim pools. The other one called "no-beach-time-yet" will list cities with current temperature bellow 30°C   We can achieve that with the input and output filtering capabilities of the MineMeld engine nodes. Let me share with you a couple of screenshots of the prototypes that will do this job:         Clone each one of these two prototypes as working output nodes and connect their inputs to the JSON miner you created. That should build a graph like the one shown in the picture.         At the time of writing this article, only one of the four cities in the feed is over 30°C.   GET /feeds/no-beach-time-yet HTTP/1.1 ... http://www.turismo.comune.parma.it/en https://ajuntament.barcelona.cat/turisme/en/ ---- GET /feeds/time-to-beach HTTP/1.1 ... http://santaclaraca.gov/visitors  
View full article
xhoms 3 weeks ago
8,087 Views
3 Replies
2 Likes
Local indicators MineMeld supports Miners where the list of indicators is stored in a local database inside the MineMeld instance. These Miners can be used to define a static list of malicious indicators or a static whitelist (for more details about using a Miner as a whitelist check the article https://live.paloaltonetworks.com/t5/MineMeld-Articles/Creating-whitelists/ta-p/72250).   You can add, delete and change the indicators stored in these Miners directly from the WebUI (the following screenshot shows an example of a whitelist, same applies to malicious indicator lists):   Prototypes for Miners of local indicators stdlib.listDomainGeneric List of domain indicators stdlib.listIPv4Generic List of IPv4 indicators stdlib.listIPv6Generic List of IPv6 indicators stdlib.listURLGeneric List of URL indicators   Uploading indicators to MineMeld Using the MineMeld API you can upload indicators to Miners of local indicators. This can be automated using the minemeld-sync.py script: https://gist.github.com/jtschichold/95f3906566b18b50cf2e3e1a44f1e785   Requirements To use the script Python 2.7.9+ is required. If you are planning to use the script from Linux or Mac OS X, python should be already available in your environment. Otherwise you can download and install python from here https://www.python.org   List of indicators The list of indicators to be uploaded should be stored in a plain text file, one line per indicator: 8.8.8.8 8.8.4.4 10.0.0.0/8 You can also add comments to each indicator, to be stored in the indicator comment attribute in MineMeld: # Google public DNS (this will be placed in the comment attribute) 8.8.8.8 # Google public DNS (this will be placed in the comment attribute) 8.8.4.4 # Private network (in the comment attribute, again) 10.0.0.0/8 You can also specify custom attributes, with the format @<attribute name>: <attribute value>: # Google public DNS (this will be placed in the comment attribute) # @direction: inbound 8.8.8.8 # Google public DNS (this will be placed in the comment attribute) # @direction: inbound 8.8.4.4 # Private network (in the comment attribute, again) # @direction: inbound # @confidence: 60 10.0.0.0/8   How to Download the script from this webpage and save it in a file called minemeld-sync.py. If you have wget installed you can use: wget -O minemeld-sync.py https://gist.githubusercontent.com/jtschichold/95f3906566b18b50cf2e3e1a44f1e785/raw Use the following command line to upload the IPv4 indicators stored in the file indicators.lst to the MineMeld instance with URL https://192.168.1.1, using the username admin and password minemeld: python minemeld-sync.py -m https://192.168.1.1 -u admin -p minemeld -t IPv4 --share-level red wlWhiteListIPv4 indicators.lst (OPTIONAL) By default old indicators are not removed from the Miner. If you prefer a full sync of the local list with the remote miner, add the options --update and --delete to the command line:   python minemeld-sync.py -m https://192.168.1.1 -u admin -p minemeld -t IPv4 --delete --update --share-level red wlWhiteListIPv4 indicators.lst   Remote certificate verification By default remote MineMeld certificate is verified using certifi package (if installed), or using the CA bundle file or CA certs directory specified via the --ca-path option: python minemeld-sync.py -m https://192.168.1.1 --ca-path /etc/ssl/certs -u admin -p minemeld -t IPv6 IPv6ListMiner my-ipv6-addresses.lst   To disable remote certificate verification use the option -k: python minemeld-sync.py -m https://192.168.1.1 -k -u admin -p minemeld -t IPv6 IPv6ListMiner my-ipv6-addresses.lst  
View full article
lmori 3 weeks ago
15,054 Views
9 Replies
2 Likes
MineMeld can be used to aggregate multiple threat intelligence feeds and extend to your Microsoft Security products via the Microsoft Graph Security API. Azure Sentinel is one of the first Microsoft Security products to ingest IOCs from the Graph Security API for use in alerting and hunting. The Microsoft Graph Security API supports the following types of Indicators of Compromise (IOCs): Email File IP address URL Domain There are three steps to connecting MineMeld to the Microsoft Graph Security API: First, you will need to create an Application in Azure Active Directory. You will assign permissions to this application to access Microsoft Graph APIs. The MineMeld Output Node will be use the credentials tied to the application you created to connect to the Microsoft Graph. You will then install the Microsoft Graph Security API extension in MineMeld Finally, you will configure the extension to connect to the Microsoft Graph via the Security API.   Azure Active Directory Configuration Login on Azure Portal – portal.azure.com Go to Azure Active Directory Navigate to Enterprise Applications – App Registrations and click on New Application Registration The Application name will not be used or surfaced as part of this integration, but we recommend you name this to Palo Alto Networks MineMeld or something similar so you can use the correct credentials when you setup your MineMeld output node. Set the Application type to “Web App / API” The Sign-on URL is not used, but you still need to put something into this field. Please note, the Application ID and Object ID will be used to configure both the threat feed in your Microsoft Graph Security API tenant and the MineMeld extension.   Under Settings go to “Required Permissions” and click on “Add” Select Microsoft Graph as an API. You can find this by typing “graph” in the search box. Under “Application Permissions “select “Manage threat indicators this app creates or owns” Click “Select” and then click “Done” Click “Grant permissions” and click “Yes” Under “App Registrations” in the app, under “Settings” go to “Keys” and create a new key with an expiration date. Click on “Save” and copy the value of the key and save it in your notes Copy the “Application ID” that you will need later Copy the Tenant ID from Azure Active Directory Properties (Directory ID)   MineMeld Configuration On MineMeld, go under “System” and “Extensions”. Glick on the Git icon Put the URL of the Github repo: https://github.com/PaloAltoNetworks/minemeld-msgraph-secapi.git Select the version (Master) and click “Install” The installation should complete shortly afterwards: Click on the enable button and confirm: The extension will activate shortly, and the API gateway will resent as part of this activation.   Integration Configuration In MineMeld, go to Config - Prototypes In the search box type “Microsoft”. Find “Microsoft_graph_secapi.output” and click on it. Click on “Clone”   Name for the node and collect it to the Input nodes you want it to use (Threat Intel feeds). NOTE: to understand the concepts of input nodes and what to connect to this, refer to Minemeld documentation Click OK Click on Commit Under Nodes, select the node you created (MicrosoftGraphSecAPI in this example) and look at the SETTINGS page. Edit Client_ID (Azure Application ID), client_secret (Azure Application secret key) and Tenant_ID (Azure Active Directory ID) Select the Microsoft services you want to share this Threat Intelligence with by clicking on “Target Product”. NOTE: As of April 2019, Azure Sentinel is the only service capable of consuming third-party threat intelligence. Testing Azure Sentinel can be used to validate this is setup correctly. Please review these instructions for turning on Threat Intelligence in Azure Sentinel. NOTE: The MineMeld extension currently specifies the Azure Sentinel service, so that is already done for you. Once you have this setup, you can review the indicators in the logs section:
View full article
3 weeks ago
1,826 Views
0 Replies
1 Like
WARNING: This article is out of date Please use follow this article to install MineMeld on a recent Ubuntu 16.04 distribution: https://live.paloaltonetworks.com/t5/MineMeld-Articles/Manually-install-MineMeld-on-Ubuntu-Server-16-04/ta-p/253336     Note for VMWare ESXi or environments with no DHCP The best way to install MineMeld on VMWare ESXi or in environments with no DHCP is using a full Ubuntu 14.04 VM: https://live.paloaltonetworks.com/t5/MineMeld-Articles/Running-MineMeld-on-Ubuntu-Server-14-04/ta-p/98454   To run a basic set up of MineMeld on VMWare desktop, you can use a simple loader that will automatically install the required packages on a stock Ubuntu 14.04 LTS cloud image and set up a basic config of MineMeld for you. This loader is packaged in a cloud-init ISO image that should be added to the standard Ubuntu OVA.   NOTE: Internet connectivity is required to setup MineMeld. First time image is booted it will automatically download all the newest required components and install them on the Ubuntu cloud image.   Video A video about installing MineMeld on VMware desktop is now availble: click here   Super fast setup To set up a demo installation, this OVA is an original Ubuntu 14.04 cloud image with an embedded MineMeld cloud-init ISO image: https://minemeld.s3.amazonaws.com/minemeld-vm-0.9.11-3build2.ova At first boot the Ubuntu 14.04 image will check for the cloud-init ISO and run the MineMeld loader.  Just download the OVA, import it into VMWare and skip to step #5 of the procedure.   1. Ubuntu 14.04 LTS cloud image MineMeld cloud-init ISO has been designed to run on Ubuntu cloud images.  Download the ova of Ubuntu trusty cloud image from here: https://cloud-images.ubuntu.com/trusty/current/trusty-server-cloudimg-amd64.ova   2. MineMeld cloud-init ISO Download MineMeld cloud-init ISO from the following link: https://minemeld.s3.amazonaws.com/minemeld-cloud-init-0.9.11-3build2.iso   3. Importing the OVA Select File > Import from VMWare Fusion and select the Ubuntu trusty OVA image. When asked do not press Finish but Customize Settings .   NOTE. If VMWare throws an error about hardware incompatibility, go to  VM > Manage > Change Hardware Compatibility and change Hardware Compatibility to the most recent version.   NOTE. If VMWare throws an error about the OVF format just press  Retry                               4. Adding the MineMeld cloud-init ISO Under VM settings, enable CD-ROM and select the MineMeld cloud-init ISO image and boot the image.   5. Accessing MineMeld VM shell Default credentials for shell access are:   Username: ubuntu Password: rsplizardspock   6. Wait for MineMeld setup At first boot the loader contained inside the ISO image will update the VM and download & setup the latest version of MineMeld. It could take some minutes.    After a successful setup you should see a message where both minemeld-engine and minemeld-web services are RUNNING.     7. Accessing MineMeld WebUI Default credentials for WebUI access are on port 443 are:   Username: admin Password: minemeld
View full article
lmori 3 weeks ago
74,756 Views
4 Replies
6 Likes
WARNING: This article is out of date Please use follow this article to install MineMeld on a recent Ubuntu 16.04 distribution: https://live.paloaltonetworks.com/t5/MineMeld-Articles/Manually-install-MineMeld-on-Ubuntu-Server-16-04/ta-p/253336     If for some reasons the supplied MineMeld cloud-init loaders for VMWare, EC2 and Azure could not work in your environment, you can fall back to the good ol' manual installation of MineMeld.   Supported distributions Ubuntu Server LTS 14.04   1. Hardening the instance First thing you should harden your new instance. MineMeld won't take off this for you. A good tutorial is this one.   2. Setting up iptables You can use the following commands to configure iptables to allow sessions on ports used by MineMeld. Port 13514/tcp is for sending syslog messages to the MineMeld instance and it is optional. Also these rules drop all IPv6 traffic, if you are running MineMeld in an IPv6 network make sure you change the suggested rules.   sudo apt-get update && sudo apt-get install -y iptables-persistent sudo iptables -A INPUT -i lo -j ACCEPT sudo iptables -A INPUT -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT sudo iptables -A INPUT -p tcp -m tcp --dport 22 -j ACCEPT sudo iptables -A INPUT -p tcp -m tcp --dport 80 -j ACCEPT sudo iptables -A INPUT -p tcp -m tcp --dport 443 -j ACCEPT sudo iptables -A INPUT -p tcp -m tcp --dport 13514 -j ACCEPT sudo iptables -A INPUT -p icmp -m icmp --icmp-type 0 -j ACCEPT sudo iptables -A INPUT -p icmp -m icmp --icmp-type 3 -j ACCEPT sudo iptables -A INPUT -p icmp -m icmp --icmp-type 8 -j ACCEPT sudo iptables -A INPUT -p icmp -m icmp --icmp-type 11 -j ACCEPT sudo iptables -P INPUT DROP sudo iptables -P FORWARD DROP sudo bash -c "iptables-save > /etc/iptables/rules.v4" sudo ip6tables -A INPUT -i lo -j ACCEPT sudo ip6tables -P INPUT DROP sudo ip6tables -P FORWARD DROP sudo bash -c "ip6tables-save > /etc/iptables/rules.v6"   3. Adding the repo GPG key Add the MineMeld repo GPG key to the APT trusted keyring:    wget -qO - https://minemeld-updates.panw.io/gpg.key | sudo apt-key add -   Double check the GPG key fingerprint, to make sure it is matching the official MineMeld GPG key (fingerprint should match characters in bold):   $ apt-key adv --fingerprint DD0DA1F9 Executing: gpg --ignore-time-conflict --no-options --no-default-keyring --homedir /tmp/tmp.W74MaAG3pI --no-auto-check-trustdb --trust-model always --keyring /etc/apt/trusted.gpg --primary-keyring /etc/apt/trusted.gpg --fingerprint DD0DA1F9 pub 4096R/DD0DA1F9 2016-07-15 Key fingerprint = E558 CE6E 3968 0F31 8F6C BFAC B401 E02E DD0D A1F9 uid Palo Alto Networks, MineMeld Team <minemeld@paloaltonetworks.com> [...]   4. Adding the MineMeld APT repo Add the MineMeld APT repo to the system list:   sudo add-apt-repository "deb http://minemeld-updates.panw.io/ubuntu trusty-minemeld main"   5. Installing MineMeld Install the MineMeld infrastructure package via apt-get. This will also automatically trigger the download of the latest MineMeld packages.    sudo apt-get update && sudo apt-get install -y minemeld rsyslog-minemeld rsyslog-mmnormalize   6. Checking if MineMeld is running Check if the 3 MineMeld services are up and running:   $ sudo -u minemeld /opt/minemeld/engine/current/bin/supervisorctl -c /opt/minemeld/local/supervisor/config/supervisord.conf status minemeld-engine RUNNING pid 3727, uptime 0:08:50 minemeld-traced RUNNING pid 3728, uptime 0:08:50 minemeld-web RUNNING pid 3729, uptime 0:08:50   7. BAM! Done! Check the Quick Tour article to get started.
View full article
lmori 3 weeks ago
90,967 Views
39 Replies
5 Likes
WARNING: This article is out of date Please use follow this article to install MineMeld on a recent Ubuntu 16.04 distribution: https://live.paloaltonetworks.com/t5/MineMeld-Articles/Manually-install-MineMeld-on-Ubuntu-Server-16-04/ta-p/253336     Click on the following link to create a new deployment of the MineMeld ARM Template in your Microsoft Azure account:   Deploy To Azure   Accessing the instance   After the template has been fully deployed, an INSTANCEWEBUI field appears in the Outputs section of the template results. The address in the field is the FQDN of the instance.     During initialization you can connect to the instance only via SSH using the credentials supplied via the template parameters. In this phase the loader is updating the Ubuntu packages, and downloading & installing the latest version of MineMeld. This phase should take a couple of minutes:   $ ssh ubuntu@mmdemo.westeurope.cloudapp.azure.com The authenticity of host 'mmdemo.westeurope.cloudapp.azure.com (40.113.113.154)' can't be established. RSA key fingerprint is 7e:8a:71:d5:c9:83:1b:a0:a6:0b:ca:c5:dc:de:75:54. Are you sure you want to continue connecting (yes/no)? yes Warning: Permanently added 'mmdemo.westeurope.cloudapp.azure.com,40.113.113.154' (RSA) to the list of known hosts. ubuntu@mmdemo.westeurope.cloudapp.azure.com's password: Welcome to Ubuntu 14.04.2 LTS (GNU/Linux 3.16.0-43-generic x86_64) [...] Ubuntu comes with ABSOLUTELY NO WARRANTY, to the extent permitted by applicable law. Initalizing minemeld. It could take some minutes, please wait .....|....   At the end of the initialization you should see a message where all the minemeld services are RUNNING:   Initalizing minemeld. It could take some minutes, please wait ..|....... DONE ! MINEMELD STATUS: minemeld-engine RUNNING pid 43691, uptime 0:00:08 minemeld-traced RUNNING pid 43692, uptime 0:00:08 minemeld-web RUNNING pid 43693, uptime 0:00:08 ubuntu@vm1:~$   Accessing the WebUI   After the initialization is completed, the WebUI can be accessed on port 443 via https with default credentials Username: admin Password: minemeld  
View full article
lmori 3 weeks ago
16,463 Views
3 Replies
WARNING: This article is out of date Please use follow this article to install MineMeld on a recent Ubuntu 16.04 distribution: https://live.paloaltonetworks.com/t5/MineMeld-Articles/Manually-install-MineMeld-on-Ubuntu-Server-16-04/ta-p/253336     To run a basic set up of MineMeld on Amazon EC2 you can use CloudFormation Launch URLs that will automatically create a new instance in your region of choice with some default settings, or create a new Ubuntu 14.04 LTS instance and specify a URL to load the user data from.   Using CloudFormation Launch URLs Use the following CloudFormation Launch URLs to spin up a new Ubuntu instance with the MineMeld loader in your region of choice:   EC2 Region CloudFormation ap-northeast-1 Launch URL ap-southeast-1 Launch URL ap-southeast-2 Launch URL eu-central-1 Launch URL eu-west-1 Launch URL sa-east-1 Launch URL us-east-1 Launch URL us-west-1 Launch URL us-west-2 Launch URL us-gov-west-1 Launch URL   Using EC2 User Data The following link contains a base64 encoded cloud-init file that can be used to automatically load MineMeld on new instances of Ubuntu 14.04 LTS. Just specify the link in the Advanced Settings while creating a new instance. https://s3-eu-west-1.amazonaws.com/minemeld-dist/0_9/minemeld-cloud-init-0.9.8-4.b64   Accessing the instance The instance can be accessed using the SSH key specified at launch. After the first boot the loader will update, download & setup MineMeld on the instance. It could take some minutes:   Welcome to Ubuntu 14.04.3 LTS (GNU/Linux 3.13.0-68-generic x86_64) [...] Ubuntu comes with ABSOLUTELY NO WARRANTY, to the extent permitted by applicable law. Initalizing minemeld. It could take some minutes, please wait ..|.......   After a couple of minutes, you should see message where both minemeld-engine and minemeld-web are RUNNING.   [...] Initalizing minemeld. It could take some minutes, please wait ..|....... DONE ! MINEMELD STATUS: minemeld-engine RUNNING pid 13086, uptime 0:00:04 minemeld-web RUNNING pid 13087, uptime 0:00:04   Accessing the WebUI After the initialization is completed, the WebUI can be accessed on port 443 via https with default credentials   Username: admin Password: minemeld 
View full article
lmori 3 weeks ago
15,739 Views
1 Reply
1 Like
The default config installed by the MineMeld loader is really simple, and can be seen in the following graph.   The 3 green nodes on the left are called Miners and are responsible for periodically retrieving indicators from 3 different feeds and push them downstream to the connected nodes using update messages. Miners are also responsible for aging out indicators: when indicators disappear from the original feed or when an indicator is considered dead, the corresponding Miner instructs the downstream nodes of removing the indicator via a withdraw message.   The central red node is a Processor node. In this specific configuration, the processor node is an IPv4 aggregator node and aggregates IPv4 indicators received from the 3 Miners and sends downstream the aggregated indicators.   The 3 yellow nodes on the right are Output nodes. These nodes receive indicators from the processor nodes and transform them into a format that could be directly consumed by external entities. In the default config the 3 output nodes translates the indicators received from the aggregator node into a format that can be consumed using the PAN-OS Dynamic Block List feature. All 3 output nodes in this graph receive the same set of indicators from the aggregator node, but each of them stores a different subset of these indicators based on the configured input filters. inboundfeedhc accepts  only indicators with confidence level > 75, inboundfeedmc only indicators with confidence level < 75 and > 50, inboundfeedlc indicators with confidence level < 50. These subset of indicators are stored into 3 different DBLs that can be used in different ways inside the PAN-OS configuration.   1. Login Default credentials are admin / minemeld     2. Dashboard The dashboard page can be used to check the overall status of the MineMeld processing engine.   In the first row you can find the number of active nodes per type, the current total number of indicators stored inside the graph and an historical chart of the total number of indicators.  The second row is dedicated to Miners, the first panel reports the total number of indicators stored in the miner nodes, and the number of indicators that have been added and aged out since engine start. The historical chart on the right represents the number of indicators added and aged out by the miner nodes in the last period of time. The third row reports the same statistics of the second row but related to Outputs.   3. Nodes The nodes view reports the status of every single active node of the graph.   4. Node status Clicking on a node in the Nodes view shows the details of the configuration of the node   the historical charts of the metrics of the node   and the connected graph where the node lives   5. Nodes and prototypes Nodes are created using libraries of prototypes. Each prototype defines the type of node and its configuration. To check the details of a prototype, just click on a node in the Nodes view and move the pointer over the prototype field   Clicking on the prototype name shows all the details about that prototype   6. PAN-OS External Dynamic Lists To use MineMeld feeds with PAN-OS External Dynamic Lists (named Dynamic Block Lists before PAN-OS 7.1), select the output node responsible for the feed in the Nodes view and copy the FEED BASE URL     In PAN-OS, under Object > External Dyanmic Lists (or Object > Dynamic Block Lists in PAN-OS before 7.1) define a new list using the output node URL   7. Whitelist indicators To whitelist a network, range or a single IPv4 address, click on wlWhiteListIPv4 under Nodes and select the INDICATORS option on the left. This list of indicators is treated as a whitelist by the aggregator.   To add a new indicator click on + and specify the IPv4 indicator. Changes will be automatically saved by pressing OK.      
View full article
lmori 3 weeks ago
67,472 Views
2 Replies
2 Likes
An easy way to use MineMeld is installing the binary packages on an Ubuntu 16.04 instance. Note: Binary packages are only available for 64 bits architectures.   Supported distributions Ubuntu Server LTS 16.04 (64 bits)   1. Hardening the instance First thing you should harden your new instance. MineMeld won't take of this for you. A good tutorial is this one.   2. Update the instance Update all the packages of the instance before installing. sudo apt update && sudo apt dist-upgrade -y   3. Setting up iptables You can use the following commands to configure iptables to allow sessions on ports used by MineMeld. Also these rules drop all IPv6 traffic, if you are running MineMeld in an IPv6 network make sure you change the suggested rules.   sudo apt install -y iptables-persistent sudo iptables -A INPUT -i lo -j ACCEPT sudo iptables -A INPUT -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT sudo iptables -A INPUT -p tcp -m tcp --dport 22 -j ACCEPT sudo iptables -A INPUT -p tcp -m tcp --dport 80 -j ACCEPT sudo iptables -A INPUT -p tcp -m tcp --dport 443 -j ACCEPT sudo iptables -A INPUT -p tcp -m tcp --dport 13514 -j ACCEPT sudo iptables -A INPUT -p icmp -m icmp --icmp-type 0 -j ACCEPT sudo iptables -A INPUT -p icmp -m icmp --icmp-type 3 -j ACCEPT sudo iptables -A INPUT -p icmp -m icmp --icmp-type 8 -j ACCEPT sudo iptables -A INPUT -p icmp -m icmp --icmp-type 11 -j ACCEPT sudo iptables -P INPUT DROP sudo iptables -P FORWARD DROP sudo bash -c "iptables-save > /etc/iptables/rules.v4" sudo ip6tables -A INPUT -i lo -j ACCEPT sudo ip6tables -P INPUT DROP sudo ip6tables -P FORWARD DROP sudo bash -c "ip6tables-save > /etc/iptables/rules.v6"   4. Adding the repo GPG key Add the MineMeld repo GPG key to the APT trusted keyring:    wget -qO - https://minemeld-updates.panw.io/gpg.key | sudo apt-key add -   Double check the GPG key fingerprint, to make sure it is matching the official MineMeld GPG key (fingerprint should match characters in bold):   apt-key adv --fingerprint DD0DA1F9 Executing: gpg --ignore-time-conflict --no-options --no-default-keyring --homedir /tmp/tmp.W74MaAG3pI --no-auto-check-trustdb --trust-model always --keyring /etc/apt/trusted.gpg --primary-keyring /etc/apt/trusted.gpg --fingerprint DD0DA1F9 pub 4096R/DD0DA1F9 2016-07-15 Key fingerprint = E558 CE6E 3968 0F31 8F6C BFAC B401 E02E DD0D A1F9 uid Palo Alto Networks, MineMeld Team <minemeld@paloaltonetworks.com> [...]   5. Adding the MineMeld APT repo Add the MineMeld APT repo to the system list and update the apt cache:   sudo add-apt-repository "deb http://minemeld-updates.panw.io/ubuntu xenial-minemeld main" sudo apt update   6. Installing nginx and redis MineMeld requires nginx and redis. We have to install them before installing MineMeld package to avoid configuration conflicts: sudo apt install -y nginx redis-server   7. Installing MineMeld Install the MineMeld infrastructure package via apt. This will also automatically trigger the download of the latest MineMeld packages.    sudo apt install -o Dpkg::Options::="--force-overwrite" -y minemeld   8. Restart We should restart the instance to make sure all the configurations are applied and all the services are started in the right order: sudo shutdown -r now   9. Checking if MineMeld is running Check if the 3 MineMeld services are up and running:   $ sudo -u minemeld /opt/minemeld/engine/current/bin/supervisorctl -c /opt/minemeld/local/supervisor/config/supervisord.conf status minemeld-engine RUNNING pid 3727, uptime 0:08:50 minemeld-traced RUNNING pid 3728, uptime 0:08:50 minemeld-web RUNNING pid 3729, uptime 0:08:50 minemeld-supervisord-listener RUNNING pid 3730, uptime 0:08:50   10. BAM! Done! Check the Quick Tour article to get started.
View full article
lmori 3 weeks ago
23,687 Views
21 Replies
2 Likes
Learn How to Safely Enable access to Office 365 using MineMeld and the new MSFT API
View full article
xhoms 3 weeks ago
53,022 Views
26 Replies
9 Likes
This article has been updated to reflect changes to the Azure AD Application registration process and to point users to a new MineMeld output node. The old node will be deprecated.   If you are not familiar with MineMeld, we recommend you start with a Quick Tour.   MineMeld can be used to aggregate multiple threat intelligence feeds and extend to your Windows Defender ATP tenant. Windows Defender ATP can ingest: IPv4 addresses File hashes URLs Domains and FQDNs There are three steps to connecting MineMeld to Windows Defender ATP: Create an application in Azure Active Directory. You will assign scopes from your Windows Defender ATP to this application, and all of the alerts tied to the threat intelligence provided will be tied to this application name. The MineMeld Miner will be associated with this application. Install the Windows Defender extension in MineMeld. Configure the extension to connect to the Windows Defender ATP tenant.   Azure Active Directory Configuration Log in to the Azure Portal (portal.azure.com). Go to Azure Active Directory. Navigate to Enterprise Applications > App Registrations > click New Application Registration.   Create a name for this application. All of the alerts tied to the threat intelligence coming from MineMeld will be attributed to this application name. We recommend calling this "Palo Alto Networks MineMeld" to avoid any confusion. NOTE: You do not need to set a redirect URI.   Click Register. From the Application page, click API Permissions.   Click Add a Permission.   Click APIs my organization uses, type “Windows” in the search bar, and select WindowsDefendertATP.   Click Application Permissions, select “Ti.ReadWrite” and then click Add Permissions.   Grant admin consent.   From the Application page, click Certificates and Secrets.   Click New Client Secret.   Copy the client secret you created.   You will also need to copy the Application ID and Directory ID.     MineMeld Configuration In MineMeld, go under SYSTEM and click the Extensions icon.   Click the GitHub icon in the lower, right-hand corner, then copy this link “https://github.com/PaloAltoNetworks/minemeld-wd-atp.git” and paste into the Repository URL field. Click the dropdown menu for Version and select “master” then click Install.   Click the checkmark to activate the extension.   The extension will activate shortly, and the empty square will signify the extension is active.   You will need to go back to the SYSTEM page and restart the API. NOTE: After the restart completes, make sure you refresh the browser page.     Setting Up the Output Node to Complete the Integration In MineMeld, click CONFIG, then click the Browse Prototype icon.   Type “windows” into the search bar to shorten the list, and select the “microsoft_wd_atp.outputBatch” node. NOTE: The “microsoft_wd_atp.output” node will be deprecated as it relies on an older API interface. Please do not use that node.   Click Clone on the top, right of the page.   Name the cloned node and add the appropriate threat feeds that you want to send to your Windows Defender ATP tenant in the INPUTS nodes section and then click OK. NOTE: To understand the concepts of input nodes and what to connect to this, refer to the MineMeld documentation on LIVEcommunity.   Click the COMMIT button in the top left of the CONFIG page.   Click NODES on the top menu and search for the node you just created. Click the node to pull up the configuration.   In Azure AD, enter the Client ID (Application), Client Secret, and Tenant (Directory) ID you copied earlier when you created the MineMeld application. NOTE: After this is done, your configuration will then be complete.     Testing To validate this is hooked up correctly, you will need to verify that an event fires if you try to access a blocked website. We recommend you create an indicator that is tied to a known good website for this, so you are not actively going to a malicious website.   Click NODES at the top and then click ADD INDICATOR   Enter in a known IP address as an INDICATOR and add it to the Input node (TYPE) you used to configure your microsoft_wd_atp.outputBatch node. Then click OK.   Wait for the indicator to be pushed to your Windows Defender ATP tenant. Then try to load that URL on a client that is running Windows Defender ATP. You should see an event fire in the Windows Defender ATP console.     Additional Information You can find out more information about this capability by reading Pushing custom Indicator of Compromise (IoCs) to Microsoft Defender ATP on the Microsoft website.
View full article
fvigo 3 weeks ago
8,460 Views
6 Replies
MineMeld is a low latency processor of indicators feeds. You can think of it as the Swiss army knife of feeds.    
View full article
lmori 3 weeks ago
73,838 Views
13 Replies
In a default config the minemeld-auto-update utility connects once per day to MineMeld auto update API to check if a new version of a MineMeld package has been released. If a new version is detected, the package is automatically downloaded and installed.   It is possible to prevent this by deleting the auto update cron job: $ sudo rm -rf /etc/cron.daily/minemeld-auto-update To reenable the auto update cron job, create the file  /etc/cron.daily/minemeld-auto-update with the following contents: #!/bin/bash /usr/sbin/minemeld-auto-update >> /var/log/minemeld-auto-update.log 2>&1  and mark it as executable.
View full article
lmori a month ago
4,290 Views
1 Reply
Each output node based on class minemeld.ft.redis.RedisSet has associated a feed accessible via the MineMeld API. The URL of the feed is shown in the node view.   Additional parameters You can use additional parameters on the feed URL to change the output format or the entry returned from the feed. You can combine multiple parameters in the same URL. Parameter Description Example (none) default format, the list of indicators is retrieved https://minemeld/feeds/feed1   Result 1.10.16.0-1.10.31.255 1.116.0.0-1.119.4.98 1.119.12.116-1.119.255.255 1.119.4.100-1.119.12.114 1.32.128.0-1.32.191.255 101.192.0.0-101.195.255.255 101.202.0.0-101.202.255.255 101.203.128.0-101.203.159.255 101.248.0.0-101.249.235.117 101.249.235.119-101.249.255.255 101.252.0.0-101.253.255.255 103.16.76.0-103.16.76.255 103.2.44.0-103.2.47.255 [...] s=<N> s=<N> retrieves entries starting from entry number N. https://minemeld/feeds/feed1?s=3   Result 1.119.4.100-1.119.12.114 1.32.128.0-1.32.191.255 101.192.0.0-101.195.255.255 101.202.0.0-101.202.255.255 101.203.128.0-101.203.159.255 101.248.0.0-101.249.235.117 101.249.235.119-101.249.255.255 101.252.0.0-101.253.255.255 103.16.76.0-103.16.76.255 103.2.44.0-103.2.47.255 [...] n=<M> n=<M> retrieves M entries from the feed. Can be combined with parameter s to select a subsect of the feed. https://minemeld/feeds/feed1?s=3&n=2   Result 1.119.4.100-1.119.12.114 1.32.128.0-1.32.191.255 tr=1 translate IP ranges into CIDRs. This can be used also with v=json and v=csv. https://minemeld/feeds/feed1?tr=1   Result 1.10.16.0/20 1.116.0.0/15 1.118.0.0/16 1.119.0.0/22 1.119.4.0/26 1.119.4.64/27 1.119.4.96/31 [...] v=json returns the indicator list in JSON format.   Note that the value of the indicator is returned only if the value flag is set in the prototype. https://minemeld/feeds/feed1?v=json   Result [ {"indicator":"1.10.16.0-1.10.31.255","value":{[...] v=json-seq returns the indicator list in JSON-SEQ format.   Note that the value of the indicator is returned only if the value flag is set in the prototype. https://minemeld/feeds/feed1?v=json-seq v=panosurl if the feed contains URL indicators, they are returned in a format compatible with PAN-OS URL EDLs https://minemeld/feeds/feed1?v=panosurl v=mwg returns the indicator list in a McAfee Web Gateway compatible format as described in https://community.mcafee.com/docs/DOC-5208 https://minemeld/feeds/feed1?v=mwg   Result type=string "iuqerfsodp9ifjaposdfjhgosurijfaewrwergwea.com" "WanaCrypt0r_Miner" "www.iuqerfsodp9ifjaposdfjhgosurijfaewrwergwea.com" "WanaCrypt0r_Miner" "iuyuip.com" "WanaCrypt0r_Miner" "oaagmx.com" "WanaCrypt0r_Miner" In the case the indicator feed is composed by IP addresses then you can modify the output type with the t=ip additional attribute   Example: https://minemeld/feeds/feed1?v=mwg&t=ip   Result type=ip "82.195.75.101" "WanaCrypt0r_Miner" "1.211.23.1" "WanaCrypt0r_Miner" "1.211.23.152" "WanaCrypt0r_Miner" "1.211.23.2" "WanaCrypt0r_Miner" "101.159.183.1" "WanaCrypt0r_Miner" "101.52.197.161" "WanaCrypt0r_Miner" "102.224.162.252" "WanaCrypt0r_Miner" "11.175.27.1" "WanaCrypt0r_Miner" v=bluecoat returns the indicator list in a BlueCoat Local List format as described in this Technical Brief document Optional attributes: cd=<category_name> (Category Default): Default Category where the indicators will be placed to ca=<attribute_name> (Category Attribute): The indicator might have an additional attribute with a list of strings describing the categories it should be listed on. Example: https://minemeld/feeds/feed1?v=bluecoat&cd=FROMAUTOFOCUS&ca=bc_category   Result define category MM_MALWARE phishyou.foobar.com end define category FROMAUTOFOCUS iuqerfsodp9ifjaposdfjhgosurijfaewrwergwea.com www.iuqerfsodp9ifjaposdfjhgosurijfaewrwergwea.com iuyuip.com oaagmx.com end v=csv returns the indicator list in CSV format.   The list of the attributes is specified by using the parameter f one or more times. The default name of the column is the name of the attribute, to specify a column name add |column_name in the f parameter value.   The h parameter can be used to control the generation of the CSV header. When unset (h=0) the header is not generated. Default: set.   Encoding is utf-8. By default no UTF-8 BOM is generated. If ubom=1 is added to the parameter list, a UTF-8 BOM is generated for compatibility.   https://minemeld/feeds/feed1?v=csv&f=confidence&f=sources|feeds&f=indicator|clientip&f=dshield_email   confidence,feeds,clientip,dshield_email 100,dshield.block,104.193.252.0/24,abuse@king-servers.com ...  
View full article
lmori ‎09-26-2019 10:42 AM
17,098 Views
5 Replies
6 Likes
An easy and powerful way of installing MineMeld is using MineMeld docker image. A docker-based installation of MineMeld can run on any Linux distribution supported by Docker and it is extremely easy to upgrade and maintain.   Overview The procedure to use MineMeld is pretty simple: Install Docker (if not already installed) - https://docs.docker.com/install/ Install & Run MineMeld The rest of the article will guide you thru installing Docker CE on RHEL 7 and run MineMeld on top of it. Only the first part, the one related to installing Docker on RHEL, is RHEL specific. The second part, the one related to MineMeld itself, is distribution independent.   Install Docker on RHEL7 Update your RHEL sudo yum update -y Remove old docker version sudo yum remove docker docker-client docker-client-latest docker-common docker-latest docker-latest-logrotate docker-logrotate docker-engine Install container-selinux package from CentOS repo sudo yum install -y http://mirror.centos.org/centos/7/extras/x86_64/Packages/container-selinux-2.107-1.el7_6.noarch.rpm Install additional packages sudo yum install -y yum-utils device-mapper-persistent-data lvm2 container-selinux Add official Docker CE repo sudo yum-config-manager --add-repo https://download.docker.com/linux/centos/docker-ce.repo Rebuild package cache sudo yum update -y && sudo yum makecache fast Install official Docker CE packages sudo yum install docker-ce docker-ce-cli containerd.io Start Docker engine sudo systemctl start docker Make sure Docker is working sudo docker run hello-world   Install & Run MineMeld Pull the latest official image sudo docker pull paloaltonetworks/minemeld Create named volumes for data and logs sudo docker volume create minemeld-logs sudo docker volume create minemeld-local Start the container sudo docker run -dit --name minemeld --restart unless-stopped --tmpfs /run -v minemeld-local:/opt/minemeld/local -v minemeld-logs:/opt/minemeld/log -p 443:443 -p 80:80 paloaltonetworks/minemeld After ~30 seconds you should see minemeld running in the container logs [ec2-user@minemeld ~]$ sudo docker logs minemeld *** Running /etc/rc.local... *** Booting runit daemon... *** Runit started as PID 7 minemeld: checking if dependencies are running... run: redis: (pid 18) 0s run: collectd: (pid 19) 0s Copying constraints Starting redis-server... Regenarating CA bundle Sep 18 14:07:31 d0b5d1fbc102 syslog-ng[20]: syslog-ng starting up; version='3.5.6' 2019-09-18T14:07:31 (35)cacert_merge.main INFO: config: {'cafile': ['/opt/minemeld/local/certs/site/'], 'dst': '/opt/minemeld/local/certs/bundle.crt', 'config': '/opt/minemeld/local/certs/cacert-merge-config.yml', 'no_merge_certifi': False} (integer) 0 Starting minemeld... /opt/minemeld/engine/0.9.64/local/lib/python2.7/site-packages/supervisor/options.py:383: PkgResourcesDeprecationWarning: Parameters to load are deprecated. Call .resolve and .require separately. return pkg_resources.EntryPoint.parse("x="+spec).load(False) 2019-09-18 14:07:32,153 CRIT Set uid to user 106 2019-09-18 14:07:32,154 WARN Included extra file "/opt/minemeld/supervisor/config/conf.d/minemeld-engine.conf" during parsing 2019-09-18 14:07:32,154 WARN Included extra file "/opt/minemeld/supervisor/config/conf.d/minemeld-supervisord-listener.conf" during parsing 2019-09-18 14:07:32,154 WARN Included extra file "/opt/minemeld/supervisor/config/conf.d/minemeld-traced.conf" during parsing 2019-09-18 14:07:32,154 WARN Included extra file "/opt/minemeld/supervisor/config/conf.d/minemeld-web.conf" during parsing 2019-09-18 14:07:32,164 INFO RPC interface 'supervisor' initialized 2019-09-18 14:07:32,164 CRIT Server 'unix_http_server' running without any HTTP authentication checking 2019-09-18 14:07:32,164 INFO supervisord started with pid 21 2019-09-18 14:07:33,167 INFO spawned: 'minemeld-supervisord-listener' with pid 60 2019-09-18 14:07:33,168 INFO spawned: 'minemeld-engine' with pid 61 2019-09-18 14:07:33,170 INFO spawned: 'minemeld-traced' with pid 62 2019-09-18 14:07:33,172 INFO spawned: 'minemeld-web' with pid 63 2019-09-18 14:07:34,322 INFO success: minemeld-supervisord-listener entered RUNNING state, process has stayed up for > than 1 seconds (startsecs) 2019-09-18 14:07:34,322 INFO success: minemeld-traced entered RUNNING state, process has stayed up for > than 1 seconds (startsecs) 2019-09-18 14:07:34,322 INFO success: minemeld-web entered RUNNING state, process has stayed up for > than 1 seconds (startsecs) 2019-09-18 14:08:03,191 INFO success: minemeld-engine entered RUNNING state, process has stayed up for > than 30 seconds (startsecs) Connect to the Web interface (https://<address>/) and use username admin and password minemeld to login   Check MineMeld Logs You can check MineMeld engine and MineMeld audit logs from outside the container. Inspect the minemeld-logs volume to grab the directory used by the Docker engine to store volume files. Your logs will be there.   [ec2-user@minemeld ~]$ sudo docker inspect minemeld-logs [ { "CreatedAt": "2019-09-18T10:07:32-04:00", "Driver": "local", "Labels": {}, "Mountpoint": "/var/lib/docker/volumes/minemeld-logs/_data", "Name": "minemeld-logs", "Options": {}, "Scope": "local" } ] [ec2-user@minemeld ~]$ sudo ls /var/lib/docker/volumes/minemeld-logs/_data minemeld-engine-stderr---supervisor-PV9ZPJ.log minemeld-supervisord-listener.log minemeld-web-stderr---supervisor-cXauy4.log minemeld-engine.log minemeld-traced-stderr---supervisor-4V3T4E.log minemeld-web.log minemeld-supervisord-listener-stdout---supervisor-T3Wfwi.log minemeld-traced.log supervisord.log   Backup MineMeld The config directory in the minemeld-local volume contains all the file needed to restore the MineMeld installation. You can backup the files by inspecting the volume and perform a backup of the config directory. [ec2-user@minemeld ~]$ sudo docker inspect minemeld-local [ { "CreatedAt": "2019-09-18T10:03:12-04:00", "Driver": "local", "Labels": {}, "Mountpoint": "/var/lib/docker/volumes/minemeld-local/_data", "Name": "minemeld-local", "Options": {}, "Scope": "local" } ] [ec2-user@minemeld ~]$ sudo ls /var/lib/docker/volumes/minemeld-local/_data certs config data library prototypes redis supervisor trace   Upgrade MineMeld To upgrade to the latest and greatest release: Pull the latest release sudo docker pull paloaltonetworks/minemeld Stop and remove the current container sudo docker stop minemeld sudo docker rm minemeld Start the container sudo docker run -dit --name minemeld --restart unless-stopped --tmpfs /run -v minemeld-local:/opt/minemeld/local -v minemeld-logs:/opt/minemeld/log -p 443:443 -p 80:80 paloaltonetworks/minemeld   Change MineMeld UI Certificate The default certificate used by MineMeld is a self-signed certificate. You should change it when running in production. To do it just map your certificate and private key on the default certificate: sudo docker run -dit \ --name minemeld \ --restart unless-stopped \ --tmpfs /run \ -v minemeld-local:/opt/minemeld/local \ -v minemeld-logs:/opt/minemeld/log \ -v /var/lib/minemeld/real-cert.crt:/etc/nginx/minemeld.cer:ro \ -v /var/lib/minemeld/real-cert.pem:/etc/nginx/minemeld.pem:ro \ -p 443:443 -p 80:80 \ paloaltonetworks/minemeld  
View full article
lmori ‎09-26-2019 10:29 AM
1,778 Views
5 Replies
1 Like
MineMeld aggregator nodes support whitelists. If an indicator is on a whitelist, the aggregator nodes will not send matching indicators to downstream nodes. Whitelists can also be shared by multiple aggregators.   Aggregator nodes created using stdlib prototypes (stdlib.aggregatorDomain, stdlib.aggregatorURL, stdlib.aggregatorIPv4Generic, stdlib.aggregatorIPv4Outbound, stdlib.aggregatorIPv4Inbound) will whitelist indicators generated by Miner nodes whose name starts with the prefix wl (lowercase).   In the following example, a whitelist Miner will be created for an IPv4 aggregator node.   1. Creating a static whitelist node In CONFIG, click + to add a new node. Specify a name starting with "wl" and select stdlib.listIPv4Generic as prototype. Enable Output and then press OK.     2. Connecting the whitelist to the aggregator In CONFIG, click on the INPUTS field of the selected aggregator. In the dialog add the new whitelist node to the list of INPUTS.   3. Commit the config Just press COMMIT in the CONFIG page.   4. Adding indicators to the whitelist In NODES, click on the new whitelist node and select INDICATORS in the menu on the left.   Click + to add new indicators. Pressing OK will automatically save the indicator and the list. It could take up to 1 minute for the new indicator to be pushed downstream to the aggregator node.  
View full article
lmori ‎09-09-2019 10:03 AM
12,864 Views
9 Replies
1 Like
The syslog miner can be used to extract indicators from logs coming from Palo Alto Networks next-generation firewall platforms.   1. Adding a syslog Miner   In CONFIG, click on the browse prototypes button and select the stdlib.syslogMiner prototype.     Inside the prototype, click on CLONE to create a new node based on the prototype.     2. Configuring the Miner   Specify a name for the new Miner, enable the OUTPUT and click OK     3. Connecting the Miner     Connect the new Miner to the inboundaggregator.     And press COMMIT.   3. Configuring syslog forwarding on PAN-OS   Please refer to the PAN-OS Administration Guide for instructions on how to configure log forwarding to a syslog server on PAN-OS (Software End-of-Life (EoL)). By defeault the syslog server on the VM listens for PAN-OS syslog messages on port tcp/13514.     4. Checking if syslog messages are received   In NODES, click on the new syslog Miner and select the STATS tab. Check the SYSLOG.PROCESSED metric, this counter is incremented every time the Miner receives a syslog message.     Log format A full list of parameters can be found in the following python script: https://gist.github.com/jtschichold/87f59b99d98c8eac1da5    Traffic logs { "src_zone": "Tap", "protocol": "tcp", "src_translated_port": "0", "dest_translated_ip": "", "app": "incomplete", "src_translated_ip": "", "log_subtype": "end", "dest_zone": "Tap", "bytes_in": "0", "generated_time": "2014/05/13 16:14:29", "serial_number": "007201000291", "type": "TRAFFIC", "action_flags": "0x8000000000000000", "packets_out": "1", "event.tags": [ "TRAFFIC", "TRAFFIC_FIELDS_6_0" ], "receive_time": "2014/05/13 16:14:29", "start_time": "2014/05/13 16:14:24", "virtual_system": "vsys1", "src_location": "US", "future_use1": "Apr 30 00:25:26 1", "future_use2": "1", "future_use3": "2014/05/13 16:14:29", "future_use4": "0", "future_use5": "0", "log_forwarding_profile": "to Panorama", "bytes": "62", "packets_in": "0", "sequence_number": "259937194", "rule": "Allow-All", "duration": "0", "repeat_count": "1", "dest_interface": "ethernet1/1", "dest_port": "9191", "category": "any", "src_port": "55869", "src_ip": "169.131.1.254", "dest_user": "pancademo\\jason.bringanti", "bytes_out": "62", "dest_ip": "10.154.14.7", "src_user": "", "src_interface": "ethernet1/1", "packets": "1", "session_id": "74404", "dest_translated_port": "0", "dest_location": "10.0.0.0-10.255.255.255", "flags": "0x64", "action": "allow" }   Threat logs { "src_zone": "Tap", "protocol": "tcp", "src_translated_port": "0", "dest_translated_ip": "", "app": "imap", "misc": "Rjf4TBoDBs389u.Emf", "src_translated_ip": "", "log_subtype": "vulnerability", "dest_zone": "Tap", "generated_time": "2014/05/13 06:55:16", "serial_number": "007201000291", "type": "THREAT", "action_flags": "0x8000000000000000", "direction": "server-to-client", "event.tags": [ "THREAT", "THREAT_FIELDS_6_0" ], "receive_time": "2014/05/13 06:55:16", "content_type": "", "virtual_system": "vsys1", "src_location": "10.0.0.0-10.255.255.255", "future_use1": "Mar 16 03:23:08 1", "future_use2": "1", "future_use3": "2014/05/13 06:55:16", "future_use4": "0", "src_user": "pancademo\\fausto.allen", "dest_translated_port": "0", "sequence_number": "1525703173", "threat_name": "Microsoft Windows Image Color Management Remote Code Execution Vulnerability(31720)", "rule": "Allow-All", "repeat_count": "1", "dest_interface": "ethernet1/1", "dest_port": "42913", "category": "any", "src_port": "143", "severity": "critical", "src_ip": "10.154.10.156", "dest_user": "", "dest_ip": "66.1.1.5", "log_forwarding_profile": "to Panorama", "url_idx": "", "src_interface": "ethernet1/1", "pcap_id": "1199806114522039943", "cloud_address": "", "session_id": "190977", "dest_location": "US", "flags": "0x80000000", "action": "alert" }   Under the Hood The following diagram depicts the message flow.   Syslog messages are received by the rsyslog deamon running on the VM. Rsyslog translates the messages into JSON and sends them to the syslog miner node inside the MineMeld engine.
View full article
lmori ‎06-28-2019 10:17 AM
15,206 Views
9 Replies
2 Likes
Introduction This document provides an example of how to configure a custom miner prototype in MineMeld in order to retrieve an external threat feed. The feed will need to be manipulated through regex expressions to only include the portions which are readable by the Palo Alto Networks firewall.     Requirements   Required Item Notes Fusion for MAC Tested with version 10.1.3 MineMeld VM Tested with V0.9.50 Palo Alto Networks Virtual Firewall Tested with VM50 PANOS 8.1.2   Use Case Diagram     Configuration MineMeld   The first step is to create a custom miner prototype.  This prototype defines the external feed location as well as any custom regex required to pull out what is necessary (and remove what is not necessary) for the firewall to read it as an external dynamic list (EDL).   After logging into your instance of MineMeld, click the config menu-bar option to see the current configured items.       In the lower right-hand portion of the page, select the icon to “browse prototypes”.     Search for a prototype miner whose type matches the type of list you wish to create (i.e. IP4, Domain, URL). In this example, I selected itcertpa.URLS in order to create a customer URL miner prototype.       Click on the desired prototype to see the details. Then select “New” to create a new prototype based on this specific miner.       Modify the NAME and CONFIG areas as needed.       In this example, I want to bring in the ransomware tracker feed located at: https://ransomwaretracker.abuse.ch/   In addition, the regex will need to be modified in order to strip the http:// and https:// from the IOC’s so the firewall EDL can read the output. The ignore_regex field will be used to ignore any lines with the # symbol (the entire line).     Click OK to save the new prototype.   The next step is to create a new miner using the new prototype.   Go to the Config area.       Click on the “eye” icon in the lower right in order to change to expert mode. Once in expert mode, a plus icon will appear allowing you to add a MineMeld node.     Select the plus, provide the new node with a name. For the PROTOTYPE drop-down, select the new prototype previously created.     Select OK to save the new miner node.   Next, create a new Aggregator node (also known as a processor node).  This node will aggregate one or more miner feeds, perform de-duplication, and prepare the data to be used by an output node.   In the Config section, select the icon to see all of the prototypes.     In the search field, type “processor” to see all of the processor prototypes. Look for one that matches the miner prototype created previously. In this example, I found stdlib.aggregatorURL. Once you find the aggregator example you wish to use, select it and then select “NEW” in the upper right-hand portion of the page to create a new aggregator node based on the one you found.   Give it a name and optionally, edit the CONFIG portion to remove any conditions that may not apply to your aggregator. In this example, I removed the area within the orange square. You may also add additional parameters depending on what you want your aggregated list to look like.     The next step is to create an aggregator node based on the new aggregator prototype just created. Go back to Config, enter expert mode, and select the plus to add a new aggregator node.   Give it a name, and for the PROTOTYPE drop-down, select the prototype just created. For the INPUTS field, select the custom miner node created in the first step.     The last node to be created is the Output node. This node will use the aggregated list and publish it to MineMeld’s internal web server so the firewall can read the final list and use it in a policy.   From the Config area, select the icon to see the prototypes. In the search field, look for “output”. Find one similar to what you want your output to look like. In this example, I used stdlib.feedGreenWithValue. Select the prototype and select “New” to create a new output based on the one selected.   Give it a name and optionally edit the CONFIG portion. In this example, I removed the portion within the orange square.       Go back to Config, enter expert mode, and select the plus to create a new output node based on the prototype just created. Give it a name, and select the output prototype in the dropdown. For the input, select the custom aggregator/processor node previously created.     Select OK to save.   You should see all three of the custom nodes created.     When ready, select COMMIT in the upper left-hand corner to save the nodes and put them to work.   To see if the list has been created, go to nodes.     Click the Output node you created and notice the FEED BASE URL link. Open the link to see the published list that the firewall will read. See a screenshot of the ransomware list below. Notice that the list no longer contains http:// or https:// references due to the regex working as expected.       The list is now ready to be consumed by the firewall.   Firewall   The firewall configuration is much easier. Browse to your Palo Alto Networks firewall and go to Objects > External Dynamic Lists and select the Add button in the lower left-hand portion of the screen.     For Type, select the appropriate type for the node type created in MineMeld. Copy the FEED BASE URL from MineMeld and paste it into Source. Optionally, Test by clicking the Test Source URL button. Click OK to save.     The final step is to use the EDL within a policy. Go to Policies > Security and add a new rule (or modify an existing rule) where you want the policy to take effect.   In the Destination tab, under Destination Address, click Add and select the EDL just created.     Commit the config.   Summary Using MineMeld is a powerful and easy way to bring in 3 rd party threat feeds based on IP, URL, and Domain. Using these feeds in your security policy is as easy as pointing the firewall to the published list and referring to the list in a policy. There are many use cases for EDL’s in both positive and negative enforcement scenarios. See the Live link below for additional ideas on incorporating EDL’s with MineMeld into your enterprise security operations.   To learn more about the free MineMeld tool: https://live.paloaltonetworks.com/t5/MineMeld/ct-p/MineMeld   To learn more about External Dynamic Lists: https://www.paloaltonetworks.com/documentation/81/pan-os/pan-os/policy/use-an-external-dynamic-list-in-policy/external-dynamic-list  
View full article
kwall00 ‎08-23-2018 08:37 AM
10,040 Views
3 Replies
1 Like
Discontinuation notice Microsoft announced a new WEB Service that will deprecate the dynamic XML document used by the miners listed in this document. A new class and corresponding set of MineMeld prototypes was introduced in version 0.9.50 to deal with the new WEB Service.   To to safely enable access to Office 365 please follow the instructions in the updated document at https://live.paloaltonetworks.com/t5/MineMeld-Articles/How-to-Safely-Enable-access-to-Office-365-using-MineMeld-Updated/ta-p/224148 Overview As customers migrate to Office 365 they find themselves whitelisting a range of App-IDs for the various workloads they might use in the Office 365 product sets, such as Skype for Business, OneNote, Exchange Online and so on. Because Microsoft publishes Office 365 over a huge range of URLs, and IP addresses, a security admin would be tempted to simply allow access in policies to a destination of ‘any’, and this gets complicated when the Office 365 App-IDs tend to have dependencies on explicitly allowing web-browsing and SSL. It would be preferable to configure external dynamic lists and reference that in our security policies, and as it happens, Microsoft dynamically publishes a fully up-to-date list of all IPs, URLs and ports used by each of the 17 components of Office 365 every hour that we can use! This article will take you through setting up the open source MineMeld utility to parse this data into EDLs for PAN-OS to consume, and creation of a couple of example security policies for your environment.   Step 1. Deploy MineMeld First, visit https://live.paloaltonetworks.com/t5/MineMeld/ct-p/MineMeld and select the article (from the top right) about installing and running MineMeld appropriate to your environment. Note, if using the VMWare desktop instructions ( https://live.paloaltonetworks.com/t5/MineMeld-Articles/Running-MineMeld-on-VMWare-desktop/ta-p/72038 ) you can go ahead with the "Super fast setup" but please download the cloud-init ISO and mount it on first boot. Assuming an IP comes via DHCP and you have internet access, your VM will automatically be updated  to the latest version of Minemeld.   Make note of MineMelds IP address (from an ifconfig) as you’ll need it for the Web UI in the next step.   Step 2. Obtain & Import Configuration MineMeld does already come with Prototypes for each of the O365 services but you would normally need to create a miner for each of these from those Prototypes, along with 3 processors and 3 outputs (one each for IPv4 addresses, IPv6 addresses and URLs respectfully). To save you the hassle we've created a configuration you can import, simply download it from https://paloaltonetworks.app.box.com/s/4ubmkgrq72a8mdd24j733ddqdgbkyvv4 and open it in a text editor.   NOTE: for a minimal config collecting all the IPv4s, IPv6 and URLs of all the O365 products download this instead: https://paloaltonetworks.box.com/s/gndwe5rzheg1ekwplxb4m3mrpcf5k41f   Browse to https://Your-MM-IP-address/ (obtained above) and sign in with the username admin and password minemeld. Next click CONFIG at the top followed by IMPORT.     This will bring up the IMPORT CONFIGURATION window. Copy and paste all the text from the .yml file you downloaded in step 2 into here and click Replace (or Append, if you have already configured this instance of Minemeld for another purpose.      Accept to replace the candidate configuration, followed by clicking the COMMIT button and waiting some time for the engine to restart.   Can't see an IMPORT button ?   This is simply because you are using an older version of MineMeld.  If you cannot upgrade for whatever reason, follow step 2a below instead.  If not, carry on to step 3.     Step 2a.  Importing configuration for an OLDER version of MineMeld only nb: Skip this step if you were able to import using the web interface as above!   SCP  the .yml file you downloaded in Step 2 to your MineMeld instance.  For example, on a Mac, run the following with the default password rsplizardspock $ scp ./office365-config.yml ubuntu@10.193.23.98: To replace the configuration of a fresh install, SSH into your MineMeld instance (again as the ubuntu user) and run the following command: $ sudo -u minemeld cp office365-config.yml /opt/minemeld/local/config/committed-config.yml Or, to append an existing configuration (ie. you have other configuration you would like to keep such as the default Spamhaus polling), run the following command or manually append the contents of office365-config.yml to the end of committed-config.yml yourself in a text editor: $ sudo -u minemeld cat office365-config.yml >> /opt/minemeld/local/config/committed-config.yml Now run the command to restart MineMeld: sudo service minemeld restart   Step 3.  Review Connection Graph and retrieve Feed Base URLs   After giving the MineMeld engine a few minutes to restart, click “Nodes” in the banner at the top of the interface and then, click any of the nodes in the list.     Then click the Graph tab (asterisk sign) to bring up the Connection Graph which should look like this:     Here you see each of the miner nodes on the left scraping Microsoft’s dynamically updated XML File (direct link for your reference: https://support.content.office.net/en-us/static/O365IPAddresses.xml ), the processor nodes that receive URLs, IPv4 and IPv6 addresses, and finally the 3 output nodes that publish a URL that your firewall can poll for an External Dynamic List (EDL).   Click each of the Output notes and make a note of the Feed Base URL.       Step 4. Consume MineMeld’s output Log into your firewall (or Panorama) and go to Objects > External Dynamic Lists (or Objects > Dynamic Block Lists if using PAN-OS prior to v7.1). Click Add and create Dynamic IP address lists and URL lists to ‘subscribe’ to each of outputs created in the previous step.  In my example below, I have created three dynamic lists matching the three Minemeld outputs above.     Step 5. Create a URL Filtering Profile   This will allow you to limit your access onto to the URLs in the O365-URLs dynamic list, which you’ll apply to your security polic(ies) allowing O365 later.  Add a URL filtering profile, and block all categories (hint: Click the top checkbox to select all items, then click the Action banner in the list, and then click “Set Selected Actions”, then block to block all categories at once).  Scroll to the bottom and allow only the external dynamic list of O365 URLs.       Step 6: Create Security Policies   Now that we have EDLs and a URL profile in place it’s time to modify/create our security policies. In the example below, we are allowing our Office 365 apps for all known users in the trust zone. The destination zone has been set to untrust zone but with the IPv4/6 lists as destination addresses.     App-IDs that you may find detected during use of Office 365 (depending on the clients and product sets being used)   activesync mapi-over-http ms-exchange ms-office365 ms-onedrive rpc-over-http soap ssl stun web-browsing webdav ms-office365 office-live office-on-demand outlook-web-online ms-lync-online ms-lync-online-apps-sharing sharepoint-online ms-lync-online-file-transfer What if there's still some O365 activity that is NOT hitting my new security policy? You may find from using a catch-all rule with logging, that some sessions are not hitting this O365 rule when they should be. The reason is because Microsoft use CDN networks, which are outside of the IPv4/v6 ranges Microsoft use, like CloudFront for some applications in O365. The following URL will allow you to confirm if this is the case and whether you need to widen your whitelist to allow for these CDNs. https://support.office.com/en-us/article/Office-365-URLs-and-IP-address-ranges-8548a211-3fe7-47cb-abb1-355ea5aa88a2   To allow access to the CDNs that do not match the security policy above, simply create a second security policy that allows from trust to untrust, from the same set of applications in the previous rule, and a destination of any. In the Service/URL category tab, insert the custom URL category from Step 5.  The FQDNs will be present in that URL category and thus match this second rule.  
View full article
sperich ‎08-01-2018 08:04 AM
56,935 Views
22 Replies
7 Likes
One of the most common use cases for MineMeld is generating feeds to be used on PAN-OS as External Dynamic Lists. Using the MineMeld powerful engine, you can create External Dynamic Lists to track on AutoFocus the IP addresses, URLs and domains used by ransomware, known APT groups and active malware campaigns. You can also create External Dynamic Lists to track the IPs and URLs used by Microsoft Office365, or used as tor exit nodes, or used by CDNs and cloud services.   In this article we provide a step-by-step guide on how to configure authentication on AutoFocus/MineMeld generated feeds. We will also cover how to configure External Dynamic Lists objects on PAN-OS 7.1 and later. In this long article there are 3 main sections: How to configure authentication on MineMeld feeds How to configure EDLs on PAN-OS 8.0+ How to configure EDLs on PAN-OS 7.1 Note for community MineMeld Even if this guide has been written for MineMeld running on AutoFocus, the same steps can be applied to the community version of MineMeld.   Note that on the community version of MineMeld feeds authentication is disabled by default. You have 2 options: enable feeds authentication and install a SSL certificate signed by a public or internal CA (strongly preferred for security reasons) leave the feeds authentication disabled 1. Configuring authentication on MineMeld feeds Configuring the authentication on MineMeld generated feeds is a simple, 3 steps process: Create a new feed user Associate at least one access tag to the new feed user Associate the same access tag to the feed you want to control Let's start with creating a new feed user. On the MineMeld user interface click on the Admin tab:   In the Admin tab, click on the circle icon on the left to select the Feeds Users tab:   Then click on the plus icon in the bottom right corner to add a new user to the list:   In the Add User dialog, specify the username of the new feed user and the password (1). These are the credentials that will be used by PAN-OS to access the feed. Once done, click on the Ok button (2 😞   Click on the Access field of the new user to specify the access tags associated with the user. The user will have access to all the feeds generated by MineMeld outputs tagged with these access tags:   Type in the Tags box (1) to associate one or more tags to the user. You can also create new tags, just type the new tag in the Tags box and press space. Click on Ok button (2) when done:   Now we have created a new feed user and associate one or more access tags to it. We should now associate at least one of these access tags to a MineMeld output to let the feed user actually have access to the feed generated by the output. Click on the Nodes tab:   Click on the output you want PAN-OS to connect to. Note The output should be based on one of the stdlib.feed* prototypes to be able to generate the feed in EDL format.     Click on the Tags field in the Status tab of the output to bring up the Tags dialog:   Add the access tag we created earlier to the Tags list (1) and click Ok (2). From now on, all the feeds users associated to this tag will be able to access the EDL generated by this output. There are 2 special tags you can associate with an output: any - any autheticated user will be able to access the generated feed anonymous - non-authenticated users will be able to access the generated feed   Take note of the URL in the Feed Base Url field of the output. This is the URL that should be configured inside the PAN-OS EDL object.   2. PAN-OS 8.0 and later - Configuring External Dynamic Lists Now that you have configured authentication on the MineMeld generated feeds, it's time to create a PAN-OS External Dynamic List to connect to the MineMeld output. The process on PAN-OS 8.0 and later has the following steps: Upload on PAN-OS the certificate of the CA of AutoFocus/MineMeld SSL certificate Create a Certificate Profile with the uploaded CA certificate for remote SSL server verification Create an EDL object using the Certificate Profile First thing, download the certificate of the CA of the AutoFocus/MineMeld SSL certificate from the following link: https://certs.godaddy.com/repository/gd-class2-root.crt   Note for community MineMeld If you have enabled authentication on feeds, you should provide and install on MineMeld an SSL certificate signed by a valid CA. The CA can be internal or public. Refer to the article How to Generate New MineMeld HTTPS Cert or to this thread (link) for the instructions.   On PAN-OS, click on the Device tab (1), select Cerificates (2) in the left bar and then click on Import (3):   Specify the Certificate Name (1), in Certificate File (2) select the CA certificate file you just downloaded (check the beginning of this section for the URL if you missed it) and click OK (3):   Now that we have uploaded the certificate, we can proceed to the next step that is creating a Certificate Profile to verify the AutoFocus/MineMeld SSL server certificate. Click on the Device tab (1), click on Certificate Profile (2) in the left bar and click on Add (3):   Specify the Name (1). We should now add the CA certificate to the list of CA certificates trusted by this Certificate Profile, click Add (2):   Select the CA certificate (1) and press OK (2):   Click OK to save the Certificate Profile:   Now we can finally create the External Dynamic List Object. Click on the Objects tab (1), select External Dynamic Lists (2) on the left bar and click Add (3 😞   In the External Dynamic Lists dialog, specify the name of the new External Dynamic List (1), select the type of indicators contained in the new External Dynamic List (2) and copy the Feed Base URL of the MineMeld output we noted down at the end of the previous section (3 😞   Now, select the Certificate Profile we created before (1). As soon as you specify the Certificate Profile, the Client Authentication section appears (2) and you will be able to specify the username and password of the feed user we created on MineMeld in the previous section. Press OK (3) to create the External Dynamic List.   Notes Test Source URL could be unreliable at times, commit the config to have a reliable check if the connectivity with MineMeld works If the Eexternal Dynamic List object is not being used in at least one policy, PAN-OS won't pull and refresh the contents   Done!! You just created a new External Dynamic List Object to point to one of the feeds generated by MineMeld. To add a second, third, ... External Dynamic Lists you don't need to do all the steps again and again but instead you can just reuse the Certificate Profile we have created.   3. PAN-OS 7.1 - Configuring External Dynamic Lists PAN-OS 7.1 doesn't support configuration of Basic Authentication for External Dynamic Lists from the Web User Interface. We should instead embed the credentials inside the URL.   Click on the Objects tab (1), select External Dynamic Lists on the left bar (2) and click Add (3):   Type the name of the new External Dynamic List (1), select the type of indicators (2) and specify the URL (3) embedding the credentials inside the URL. Example: if the username is edluser and password test123 the URL would be https://edluser:test123@<minemeld hostname>/feeds/<feed name>. Press Ok (4 😞   Done!! You just created a new External Dynamic List Object to point to one of the feeds generated by MineMeld.  
View full article
lmori ‎12-07-2017 09:26 AM
28,008 Views
10 Replies
2 Likes
MineMeld was conceived as a Threat Indicator Sharing platform. Its main goal is as  a tool to help users consume all sorts of threat indicators from diverse sources, aggregate them, age them, then finally make them actionable by featuring very flexible output nodes. Over time, MineMeld has incorporated Data API's and local storage for indicators, which has extended its range of use cases. In this article in the Using MineMeld... series, we'll cover the use case of MineMeld as a Incident Response Platform.   The following bullets describe the need for Incident Response we'll solve in this article: You own a VirusTotal Intelligence account that you use to perform hunting for malware samples with YARA rules. The corresponding VirusTotal notification service provides you with fresh malware incidents (SHA256 hashes) seen on its community of users that matches the YARA rules you've provided. You have deployed PA-Series Next Generation Firewalls for segmentation (either Internet Gateway or Datacenter) with WildFire subscription that unmask unknown malware crossing your network (incident) in less than 5 minutes. A set of endpoints is secured with TRAPS (effectively blocking zero day malware). But you also take care of a large base of endpoints protected by legacy AV solutions that do not provide protections against zero day malware. But that same AV solution can block execution of binaries provided someone could keep a list of SHA256 hashes belonging to malicious files that should be blacklisted. So you want MineMeld to provide a response keeping an updated output node with malware hashes coming both from VT and WildFire incidents that will be consumed by the legagy AV solution. This way you buy time to execute the migration of the large installed base to TRAPS. Step 1. Deploy MineMeld First, visit   https://live.paloaltonetworks.com/t5/MineMeld/ct-p/MineMeld   and select the article (from the top right) about installing and running MineMeld appropriate to your environment. Note, if using the VMWare desktop instructions (https://live.paloaltonetworks.com/t5/MineMeld-Articles/Running-MineMeld-on-VMWare-desktop/ta-p/72038) you can go ahead with the "Super fast setup" but please download the cloud-init ISO and mount it on first boot. Assuming an IP comes via DHCP and you have internet access, your VM will automatically be updated  to the latest version of Minemeld.   Make note of MineMelds IP address (from an ifconfig) and login from your browser (defaults to username: admin / password: minemeld)   Step 2. Create the MineMeld Graph At the end of this article you'll find copy&paste ready snipets. But we highly encourage you to follow the steps to get used to MineMeld's WEB UI features and procedures.   Let's start with a blank configuration. The following is the summary of steps we'll take: Clone the out-of-the-box stdlib.feedHCGreenWithValue prototype into our graph as output node. Clone the out-of-the-box stdlib.aggregatorSHA256 prototype into our graph as aggregator node. Create a new prototype based on the virustotal.notifications one to be able to share VT intelligence into the green share level. Create a new prototype based on the stdlib.localDB  one to be able to store WildFire events into the green share level. Link the nodes to form a connected graph.   To start we must reach the prototype library that is available by navigating to small icon in the down left part of the Config Tab.     In this example we'll trust all indicators coming from VirusTotal and WildFire. This means that any SHA256 hash delivered by these miners will have a value of 100 in the confidence attribute. So we can just clone the  stdlib.feedHCGreenWithValue prototype that will accept all indicators with confidence > 75 shared in the green level.         We can, as well, clone the  stdlib.aggregatorSHA256 from the prototype library into our graph.         The VirusTotal miner prototype in the library is still tagged as experimental which means that all indicators used by it will be placed in the red share level. As our output node only accepts indicators in the green share level, we must create a new prototype using the one in the library as a base but chaging the share level attribute value.       Note that the prototype in the library shares in the red channel. Click on "new" to create a new prototype that shares in the green level instead.     Now it is time to clone this recently created prototype into our graph.         We, finally, need a miner node that could receive async events (WildFire logs from a PANOS device) and store them into a local database. This miner must also provide aging for the indicators so they are removed from the list at a given expiration time. This is what the class minemeld.ft.localdb.Miner exposed through the stdlib.localDB prototype is all about.   As in the case of the VirusTotal prototype, this node is tagged as experimental which means we must first create a new prototype to share indicators in the green level before we can clone it into our graph.               At this moment our configuration should look like the one in the following picture. Note that the nodes are not linked. We can bind them together by clicking on the input cell of both the aggregator and the output node.     Once all nodes are conneted we just need to commit the configuration to get our working graph.     The graph can be shown by clicking on any node in the Nodes Tab and selecting the graph view.      Step 3. Configure authentication in the output node A good practice is to use authentication in the output nodes so only approved clients can grab indicators from it. MineMeld features a very flexible feed authentication mechanism: The administrator can create "feed users" that can latter be attached as "tags" to the output nodes. A given output node can host multiple tags which means it will accepts connections from more that one customer.   First of all let's access the feed user panel available in the Admin main tab. Once there, a new feed user can be added by clicking on the plus icon.        The feed user must then be attached to the output node. That can be achieved by clicking on the output node from the Nodes main Tab. In the node's status pannel you must click on the TAG row to attach the feed user authentication to that node.               We are all set for now. We just need the miners to start gathering indicators for us.   Step 4. Connect MineMeld with VirusTotal You might have noticed the small error icon besides the VirusTotal miner node. That error indicates that the node is not fully configured. A mandatory attribute is not yet provided: The VirusTotal Intelligece API Key.   The key can be obained from the VirusTotal page under the profile settings. Note, please, that the API Key must come from a Virus Total Account with Intelligence access permissions. If you're not familiar with VirusTotal malware hunting features please visit https://www.virustotal.com/intelligence/help/malware-hunting/ before using the VT miner.       Once you have copied the key to your clipboard just paste it into the API KEY field available in the status tab of the virustotal node panel.       No commit is needed. Just force a polling by clicking on the icon at the end right of the LAST RUN field. If everything is right the indicators field will be updated with a non zero value. If you want to take a look to these indicators just navigate to the output node an click on the FEED BASE URL to get a plain text version of the list.     Step 5. Configure WildFire incident forwarding The last step in our mission today is to capture WildFire events from our PANOS device to extract SHA256 indicators out of them. To achieve this we'll leverage the HTTP Forwarding feature introduced with PANOS 8.0.   The following is our theory of operations: PANOS generates a WildFire event log with multiple fields. One of them is the "file_digest" that contains the SHA256 of the recently detected malware. A log forwarding object is attached to the security rule that features WildFire inspection The log forwarding object accepts all events (no filter needed) in the WildFire category, formats them in a way that the miner node can consume and forwards it using HTTPS to MineMeld. MineMeld's default distribution includes a self signed certificate. For the HTTPS protocol to work you must: Issue a certificate for MineMeld's web server from a PKI that is in the trust list of the PANOS Device. Alternatively you can use the PANOS Device as a PKI and generate the MineMeld certificate from it. Follow the instructions in https://live.paloaltonetworks.com/t5/MineMeld-Discussions/Minemeld-SSL-Certificates/m-p/112938 to import that certificate into your MineMeld instance. First of all we need a new HTTP Server Profile for the MineMeld. We must provide: A name An address or FQDN Protocol and Port Method Credentials (we can use any MineMeld admin user credential. Feed user credentials won't work)     The localDB miners's API accepts payloads either in JSON or PlainText format. In any case the type and indicator fields are the only mandatory ones. Optionals fields are: share_level: to override miner's share_level value. A given threat intel source might want to specify the share level of a given indicator. ttl: the amount of seconds the indicators should be kept into the store. A null ttl value means the indicator won't ever expire. In this example we'll use a 1h ttl so you can verify how the indicators are aged out by the miner. any other attribute will just be attached to the indicator to enrich it.   A couple of comments in the payload format shown in the picture above: The url format must be that exact literal just changing the text in red to match the miner's node name we want to forward the indicators to. In our example /config/data/WildFireEvent_indicators/append A Content-Type header with the value application/json is needed for the MM API to accept json data. The parameter t must be equal to localdb to forward the received data to a localdb type of miner. The optional parameter h can contain a node name. If it exists the MineMeld engine will trigger a node "sync request" to the named node after the data has been processed. In this case we want the indicator captured to be automatically flushed down the graph path so we request a sync of the WildFireEvent node. Note the $filedigest variable in the payload field. This tells the PANOS forwarding engine to replace that space in the payload with the corresponding field value in the event log. The next step is to create a new log forwarding object to feature this collection of HTTP server and payload format we've just defined.     Finally we must attach this log forwarding object to the security policy rules that features WildFire inspection.     If you do not want to wait for WildFire incidents just trigger a new one by downloading the WildFire test PE file. Once you get your incident just confirm that it is available in the MineMeld's output feed.               MineMeld's feed output node, by default, provides the content in a plain text format. But you can modify this behavior with additional attributes as explained in Parameters for the output node to easy the consumption by third parties. And, why not, contribute to the MineMeld OpenSource project with new output nodes like, McAfee DXL (TIE) or Tanium.   Annex 1: the prototypes Local Prototypes used in this use case (/opt/minemeld/local/prototypes/minemeldlocal.yml)   author: minemeld-web description: Local prototype library managed via MineMeld WebUI prototypes: localDBGreen: class: minemeld.ft.localdb.Miner config: attributes: confidence: 100 share_level: green interval: 3600 description: 'list of indicators. Use a name starting with "wl" to create a whitelist for an aggregator ' development_status: EXPERIMENTAL indicator_types: - any node_type: miner tags: - ConfidenceAny - ShareLevelAny virusTotalShareGreen: class: minemeld.ft.vt.Notifications config: age_out: default: 365d interval: 7200 sudden_death: false attributes: confidence: 100 share_level: green description: 'Miner for VirusTotal Intelligence Notifications feed. ' development_status: EXPERIMENTAL indicator_types: - md5 - sha256 - sha1 node_type: miner tags: - ConfidenceHigh - ShareLevelRed   Graph configuration (Config -> Import)   nodes: VirusTotal: inputs: [] output: true prototype: minemeldlocal.virusTotalShareGreen WildFireEvent: inputs: [] output: true prototype: minemeldlocal.localDBGreen sha256agg: indicator_types: - sha256 inputs: - VirusTotal - WildFireEvent node_type: processor output: true prototype: stdlib.aggregatorSHA256 sha256out: indicator_types: - any inputs: - sha256agg node_type: output output: false prototype: stdlib.feedHCGreenWithValue Annex 2: the API In MineMeld 0.9.42 we have introduced a new type of Miner (stdlib.localDB) exposing the HTTP-based API used in this article. In the following examples we assume that you have created a localDB Miner called FooBar.   A2.1: Push new indicators via JSON You can push new indicators using JSON. The ttl attribute is the expiration time in seconds of the indicator. If omitted the default config value of the Miner will be used. Only the indicator and type attributes are mandatory. curl -XPOST -H "Content-Type: application/json" -u admin:minemeld "https://192.168.55.150/config/data/FooBar_indicators/append?h=FooBar&t=localdb" -d ' { "indicator": "8.8.8.8", "type": "IPv4", "comment": "usual Google DNS Public IP", "share_level": "green", "confidence": 100, "ttl": 3600 }'  A2.2: Delete indicators via JSON You can use the same API call to delete an indicator, by setting the ttl attribute to 0: curl -XPOST -H "Content-Type: application/json" -u admin:minemeld "https://192.168.55.150/config/data/FooBar_indicators/append?h=FooBar&t=localdb" -d ' { "indicator": "8.8.8.8", "type": "IPv4", "ttl": 0 }' A2.3: Push new indicators via text format You can also use a simple text format to push a list of indicators: Indicators are divided by a blank line The first line of an indicator is the type of the indicator The second line is the indicator itself After you can specify the attributes, where a line is the attribute name and the line after is the attribute value Example: curl -XPOST -H "Content-Type: application/text" -u admin:minemeld "https://192.168.55.150/config/data/FooBar_indicators/append?h=FooBar&t=localdb" -d ' URL https://www.google.com comment test comment confidence 80 share_level green ttl 1800 IPv4 8.8.8.8 comment usual Google DNS Public IP share_level green confidence 100 ttl 86400 ' A2.4: Delete indicators via text format You can use the same API call to delete an indicator, by setting the  ttl  attribute to  0 : curl -XPOST -H "Content-Type: application/text" -u admin:minemeld "https://192.168.55.150/config/data/FooBar_indicators/append?h=FooBar&t=localdb" -d ' URL https://www.google.com ttl 0 IPv4 8.8.8.8 ttl 0 '  
View full article
xhoms ‎12-04-2017 10:41 AM
23,268 Views
15 Replies
1 Like
Overview In this article we introduce the classes available in the base MineMeld distribution and how to use them to create a managed services offering for your customer base. Either to differentiate yourself from these pure resellers that do not add any value or to build a new income source.   Installing your MineMeld instance If this is your first time with MineMeld then you better read the Getting Started article and deploy your first MineMeld instance following the article from the following list that better suits your needs. Running MineMeld on VMWare desktop Running MineMeld on Amazon EC2 Running MineMeld on Microsoft Azure Manually install MineMeld on Ubuntu Server 14.04 Base offering In this case example we will start with a basic offering consisting on a high-risk IPv4 feed that any customer should block at the Internet Gateway. That feed will consist on the aggregation of the following three standard miners: dshield.block spamhaus.DROP spamhaus.EDROP In other words, we will create the following graph.     To reach to this configuration point you just have to import the following configuration snippet in the configuration section of the WebUI.   nodes: dshield_blocklist: output: true prototype: dshield.block myBaseOffering: inputs: - spamhaus_EDROP - spamhaus_DROP - dshield_blocklist output: true prototype: stdlib.aggregatorIPv4Generic spamhaus_DROP: output: true prototype: spamhaus.DROP spamhaus_EDROP: output: true prototype: spamhaus.EDROP freeBaseOffering: inputs: - myBaseOffering output: false prototype: stdlib.feedHCGreen         All MineMeld feeds are exposed using a basic authentication mechanism. By default only the MineMeld admistrator user and password can be used to grab the feeds contents. Note, please, that if you check the feed URLs from a WebBrowser that is already connected to the MineMeld WebUI then you won't experience the authentication request because your browser has already been authenticated and do have a session cookie.   To implementa a granular control of feed content by customer or customer type you must create a "feed user" and give it a password in the Admin section of the WebUI.           Once the feed user has been created you must assign it an authentication tag and attach that tag to the output node.         Now it is time to commit the configuration changes.     To check that everything is fine so far just validate the feed is working by attempting to access its data with the corresponding user and password (basic authentication).     user$ curl https://yourBrand:xxxx@minemeld/feeds/freeBaseOffering 185.35.62.0-185.35.62.255 80.82.77.0-80.82.77.255 1.10.16.0-1.10.31.255 1.32.128.0-1.32.191.255 101.192.0.0-101.195.255.255 101.202.0.0-101.202.255.255 101.203.128.0-101.203.159.255 101.248.0.0-101.249.255.255 101.252.0.0-101.253.255.255 102.176.0.0-102.177.255.255 102.181.0.0-102.181.255.255 103.16.76.0-103.16.76.255 ...   Now it is time to configure the Internet Gateway to block any packet coming from or going to any IP address belonging to these ranges. In the following example I will configure an IP External Dynamic List object linked to the feed we have just created and a couple of Policy Based Forwarding rules to discard traffic when these IP's are involved.         If your customer wants to be able to monitor the activity of traffic matching the EDL then you can use Security Policies with drop action instead of Policy Based Forwarding rules.       Adding intelligence to a premium customer What about suggesting a specific SMB customer to subscribe to a premium service where other intelligence data is combined into his reputation fee?   We will continue with our example adding a new miner (the TOR exit nodes) to our instance and modifying the graph in such a way that only a given specific premium customer is allowed to consume this extended feed.       Try to clone nodes from the standard library to create the graph or, if you want to go straight to the point, just import the following configuration snippet into you configuration (use the “Append” button)   nodes: torExitNodes: inputs: [] output: true prototype: blutmagie.tor_exit_nodes premiumOffering: inputs: - myBaseOffering - torExitNodes output: true prototype: stdlib.aggregatorIPv4Generic premiumCustomerA: inputs: - premiumOffering output: false prototype: stdlib.feedHCGreen     And remember to create a new feed user for this premium customer and attach its corresponding authentication tag to the output feed.         Reputation Managed Service The next step in your offering could be to provide a managed service to customize the reputation feed with any arbitrary source data. To do that we’ll add a “local list” (prototype: stdlib.listIPv4Generic) miner for one specific premium customer so the field engineers have a bucket in which any indicator requested by the customer could be placed.       Just append the following configuration snippet to append the two remaining nodes to complete the graph.   nodes: bucketCustomerA: inputs: [] output: true prototype: stdlib.listIPv4Generic customerA: inputs: - premiumOffering - bucketCustomerA output: true prototype: stdlib.aggregatorIPv4Inbound   Note that, at this moment, the premiumCustomerA input will still be attached to the premiumOffering aggregator node. So we must change it to the recently created aggregator node called customerA.       This new bucketCustomerA node provides a way to add new indicators to the managed customer feed. The easiest way is to use the node’s WebUI feature.             MineMeld's API provides you with a way to upload indicators into a list to deal with bulk operation. A nice implementation is available in minemeld-sync tool available in GitHub.   Other Managed Services ideas I hope this article gave you interest in exploring ways to leverage MineMeld to add value to your NGFW deals and, eventualy, use it as a revenue stream.   The following is a list of ideas you could evaluate to extend your value proposition: Automatic provisioning of IOC's to customer's Internet Gateways when a major malware campaign is published (URL's, Domains and IP's). Offering yourself as a bridge between the local security /vertical market CSIRT and the interested customers. Managed service to limit YouTube video exposure by channel or playlist ( more info in Filtering YouTube videos to only approved playlists ) Automatic enforcement of SOC alerts
View full article
xhoms ‎12-04-2017 04:20 AM
5,805 Views
0 Replies
7 Likes
Overview In some cases you might face the need to create a policy rule in a Palo Alto Networks next generation firewall that targets a large list of IP addresses that shares a common schema.   For example: All printers in a set of branch office networks that happens to be the ".7" in a collection of subnets where the third byte is a variable: "192.168.x.0/24" All even networks in the collection "10.10.x.0/24" Using a set of wildcard masks could ease the description of these lists. For example, the first case fits within the wildcard mask 192.168.0.7/255.255.0.255 and the second example in the wildcard mask "10.10.0.0/255.255.1.0"   A miner extension available at https://github.com/PaloAltoNetworks/wildcardip-miner can be used to generate these lists of IP's from user provided wildcard masks.   In this article you'll find all the steps needed to deploy MineMeld and configure a Palo Alto Networks NGFW using the wildcard extension miner.   Step 1. Deploy MineMeld First, visit   https://live.paloaltonetworks.com/t5/MineMeld/ct-p/MineMeld   and select the article (from the top right) about installing and running MineMeld appropriate to your environment. Note, if using the VMWare desktop instructions (https://live.paloaltonetworks.com/t5/MineMeld-Articles/Running-MineMeld-on-VMWare-desktop/ta-p/72038) you can go ahead with the "Super fast setup" but please download the cloud-init ISO and mount it on first boot. Assuming an IP comes via DHCP and you have internet access, your VM will automatically be updated  to the latest version of Minemeld.   Make note of MineMelds IP address (from an ifconfig) and login from your browser (defaults to username: admin / password: minemeld)   Step 2. Add the wildcardip extension MineMeld introduced support for external extensions in the version 0.9.32. You can install them either localy or from a GitHub repository. To use the GitHub (recommended) option you must access to the repo URL available in its main page ("Clone or Download" button)     The URL for the repo is https://github.com/PaloAltoNetworks/wildcardip-miner.git. Use this URL in the "Install Extension From Git" option in MineMeld's WebUI (accessible through the "System" menu)     Once the extension is successfully downloaded, the next step is to activate it.     Step 3. Creating a wildcardip node The wildcard extension provides an example prototype that must be customized for our needs. The following is the sequence of tasks needed to add a node: Create a "New" prototype using the example provided one. Configure this new prototype with the needed list of wildcard masks "Clone" the just created prototype to convert it into a working node. All these tasks are performed under the "Configuration" main menu in MineMeld.       Once we reach the prototype library, use the "search" field to find the wildcard prototype provided by the extension.       Click over the prototype to display its contents and to be able to access the "new" and "clone" operations. In this case we need to create a "new" prototype that suits our needs.       Clicking on "new" will show the YAML prototype editor.       This example creates a wildcard that will generate IP address objects that will match addresses .1 to .15 in a list of networks 192.168.[0-255].0. The meaning of the rest of the attributes in the prototype is: age_out: It describes the "age out policy" of indicators for this node. In this case: default = null : indicators are not aged out interval = 86400 : check for aged out indicators once a day sudden_death = true : remove indicators if they're not longer in the input list interval: the amount of seconds between polls on this miner. We used 86400 to regenerate the list once a day. But it is safe to use larger values. max_entries: a wildcard-class specific attribute that allows the user define a safety limit on the entries that should be generated by this node. The miner will fail to generate entries if the provided wildcard masks extend over this limit. wildcard_list: a list of wildcard masks to be expanded. Verify, please, the YAML syntax (using TAB of each entry in the list) Once the new prototype is added to the library, we can generate a node out of it using the "Clone" procedure.         Next task is to attach an output node to our recently created wildcard miner. Take note of the name provided to the miner because it will be used in the output creation.     Clone the stdlib.feedHCGreen prototype from the list as a node and configure the willdcard miner as its input.     At this moment, configuration should look like the following screen capture. Commit the new configuration.     Step 4: Enable feed authentication Once the configuration is commited a new graph will be available in the "Nodes" main menu.     The number of indicators (256) corresponds is the result of the expansion of the wildcard mask provided in the prototype (192.168.0.0/255.255.0.240)   The details panel of the output contains the URL link to the feed. Clicking in it will provide the resulting list of the wildcard expansion.       At this moment in time is important to note that you have been able to see the feed contents because the browser have a session cookie with the MineMeld WebUI. You can verify it by opening a new "incognito mode" window in your browser and attempting to access the URL from there. You should receive an authentication error in this case.   Before we keep going on, you must replace the original MineMeld SSL self-signed certificates with a valid ones issued by your local CA. The article How to Generate New MineMeld HTTPS Cert explains how to use a NGFW or Panorama as a local CA.   MineMeld feed authentication framework contains the following elements: Feed User: A user/password pair with a list of "Access Tags" it belongs to. Access Tag: A label that can be attached to a feed to limit its exposure to only the users belonging to that tag. Feed Users are created from the "Admin" main menu.     And access tags created clicking on the user's "Access" cell.     Next task is to attach this recently created access tag to the feed. The way to do this is by navigating to the feed node and clicking in its access cell.     All these changes apply at runtime. No commit is needed. We're ready for the next step.   Step 5: PANOS Configuration NGFW from Palo Alto Networks feature a dynamic object called "External Dynamic List" (EDL) that can be used to consume the feed generated by MineMeld.   The following components are needed to successfuly complete the EDL configuration: The URL of the feed. A valid user/password pair that belongs to the access tag of that given feed. Certificate Profile including the CA that issued MineMeld's SSL certificate (for PANOS 8.0) NGFW Policy referencing the EDL (PANOS won't try to download the EDL until it is consumed somewhere)     User and password fields will not appear until you select one certificate profile. For PANOS 7.1 or before you must use the https://<username>:<password>@url syntax instead.     Final EDL configuration should look like the following screenshot.     Once configuration is commited, list entries should appear (provided that you are consuming this EDL anywhere in your NGFW configuration)     Annex A: Advanced MineMeld wildcard use case Imagine that you're asked to extend the previous case removing the ".5" nodes from the list. This could be described like substract the list created by "192.168.0.5/255.255.0.255" from the list created by "192.168.0.0/255.255.0.240".   This can be achieved easily with the "whitelist" feature of the stdlib.aggregatorIPv4Generic processor. We just need to add a new wildcard miner with the mask "192.168.0.5/255.255.0.255", give it a name starting with "wl_" (whitelist) and combine it with the miner from the previous exercice.     If you take a look to the feed output, you'll notice the "aggregator magic". It now describes the address spectrum as a list of ranges. For each network you have two ranges: 192.168.x.0-192.168.x.4 and 192.168.x.6-192.168.x.15    
View full article
xhoms ‎11-27-2017 04:45 AM
8,715 Views
3 Replies
1 Like
User credentials for MineMeld UI are stored in /opt/minemeld/local/config/api/wsgi.htpasswd in the standard Apache2 htpasswd format.   1. Installing htpasswd utility It's not installed by default, it is contained in the apache2-utils package. $ sudo apt-get install apache2-utils   2. Changing UI admin password $ sudo htpasswd /opt/minemeld/local/config/api/wsgi.htpasswd admin New password: Re-type new password: Updating password for user admin   3. Restarting minemeld-web To enforce the new credentials the MineMeld Web API should be restarted. This doesn't restart the MineMeld engine, just the frontend component. $ sudo -u minemeld /opt/minemeld/engine/current/bin/supervisorctl -c /opt/minemeld/supervisor/config/supervisord.conf restart minemeld-web minemeld-web: stopped minemeld-web: started    NOTE:  To add a new user to the UI just use the htpasswd command at step 2 with a new username instead of admin.
View full article
lmori ‎07-02-2017 11:59 PM
12,898 Views
7 Replies
Using MineMeld TAXII output nodes and IBM QRadar Threat Intelligence app, it is possible to populate IBM QRadar reference sets with Threat Intelligence indicators processed by MineMeld. Reference Sets can then be used in IBM QRadar rules to detect suspicious activities   After installing IBM QRadar Threat Intelligence app (available on IBM QRadar App Exchange), follow this procedure to connect IBM QRadar to MineMeld: Create one or more TAXII DataFeed Output nodes in MineMeld Define authentication for TAXII DataFeed nodes (optional) Upload CA certificate to IBM QRadar Threat Intelligence app Configure one or more reference sets on IBM QRadar (optional) Configure MineMeld TAXII Feeds on IBM QRadar Threat Intelligence app 1. Create TAXII DataFeed Output nodes in MineMeld Use stdlib.taxiiDataFeed prototype to instantiate one or more output nodes. Each output node becomes a new TAXII data feed IBM QRadar can grab indicators from. In this picture all the taxiiKnownCampaigns* nodes are TAXII DataFeed nodes.   2.  Define authentication for  TAXII DataFeed nodes This step is required only if you have enabled authentication for feed access.   Note for MineMeld on AutoFocus Authentication for feeds is automatically enabled in MineMeld on Autofocus.   2.1. Define a feed user Under Admin > Feeds Users create a new user and associate an access tag to it. The new feed user does not have access to the Admin WebUI, but only to feeds tagged with at least one of the tags listed in the ACCESS field. In the following picture SOC_QRadar user has access only to feeds tagged with siem.   2.2. Configure tags on TAXII DataFeed nodes Under Nodes select the TAXII DataFeed nodes and add the access tag.                                   3. Upload CA certificate to IBM QRadar TI app IBM QRadar Threat Intelligence app requires a valid certificate on the TAXII server. If the certificate on your MineMeld instance is signed by a private CA or a CA not known to IBM QRadar Threat Intelligence app, you have to upload the certificate on the app. The certificate should be in PEM format and the extension of the file should be pem.   Note for MineMeld on AutoFocus Download the GoDaddy Class2 Root certificate here https://certs.godaddy.com/repository/gd-class2-root.crt, change the extension to pem and upload it to the app.                                                       4. Configure one or more reference sets on IBM QRadar This step is optional. To keep MineMeld indicators separated from other sources you can define a new Reference Set for each MineMeld DataFeed.                                                                   5. Configure TAXII Feeds In the IBM QRadar Threat Intelligence app, select Add TAXII Feed.   5.1. Configure TAXII server parameters In TAXII Endpoint set https://<minemeld address>/taxii-discovery-service If feeds authentication is enabled on MineMeld, select HTTP Basic in Authentication Method and set Username and Password of a MineMeld feed user with access to the TAXII DataFeed. Then click on Discover.                                       5.2. Select the collection In the next dialog select the Collection, and set the appropriate Observable Type - that is the type of the indicators in the MineMeld TAXII DataFeed. Each MineMeld TAXII DataFeed node is seen as a separate Collection.   5.3. Select the target reference set In the next dialog, select the target reference set.   5.4. Save & Poll Click on Save.     In the TAXII Feed list, click on Poll Now to retrieve the indicators from the datafeed.  
View full article
lmori ‎02-16-2017 03:50 AM
12,097 Views
7 Replies
1 Like
You can use PAN-OS log-links feature to link your PAN-OS WebUI to MineMeld and search MineMeld logs or manage False Positive directly from PAN-OS logs view.   1. Searching MineMeld logs from PAN-OS Webui In PAN-OS CLI use the following commands to create log-links for searching the source IP or destination IP of the session in MineMeld logs. Replace <minemeld-address> with the address of your MineMeld instance.   admin@ngfw> configure admin@ngfw# set deviceconfig system log-link "MineMeld Search Src" url "https://<minemeld-address>/#/logs?q=indicator:{src}" admin@ngfw# set deviceconfig system log-link "MineMeld Search Dst" url "https://<minemeld-address>/#/logs?q=indicator:{dst}" admin@ngfw# commit [...] 1.1. Demo     2. Managing False Positives from PAN-OS WebUI In PAN-OS CLI use the following commands to create log-links for addingthe source IP or destination IP of the session to MineMeld whitelists. Replace <minemeld-address> with the address of your MineMeld instance.   admin@ngfw> configure admin@ngfw# set deviceconfig system log-link "MineMeld Src FP" url "https://<minemeld-address>/#/indicator/add?indicator={src}&indicatorType=IPv4" admin@ngfw# set deviceconfig system log-link "MineMeld Dst FP" url "https://<minemeld-address>/#/indicator/add?indicator={dst}&indicatorType=IPv4" admin@ngfw# commit [...] 2.1. Demo      
View full article
lmori ‎02-09-2017 06:31 AM
3,038 Views
0 Replies
If you are using your Palo Alto Networks firewall as a trusted root CA, you can generate a web server certificate for MineMeld to replace the self-signed one.   Start Inside WebGUI  Steps: Go to your Palo Alto Network Firewall or Panorama WebGUI Device > Certificate Management > Certificate At the bottom of the screen, click Generate, to create a new certificate. Ensure that it is signed by the firewall by clicking "Certificate Authority". Export the pem file with the private key by clicking the certificate you want to export and clicking Export at the bottom of the screen. Then use "Base64 Encoded Certificate (PEM)" and also select "Export private key" and click OK. Now to the CLI Steps: Now copy the cert to minemeld with the command: > scp cert_minemeld.pem ubuntu@minemeld Now, log into MineMeld via ssh with the command: ssh ubuntu@minemeld Now backup the current certificates in case you need to revert back to them if something goes wrong [minemeld ~]$ sudo mv /etc/nginx/minemeld.cer /etc/ngnix/minemeld.cer-orig [minemeld ~]$ sudo mv /etc/ngnix/minemeld.pem /etc/ngnix/minemeld.pem-orig The pem file that you generated will have both the private and public key so you need to split the two. This is how I did it, you may have a better way.  The file is readable so you can copy and paste the sections into two different files or use the CLI commands: [minemeld ~]$ cat cert_minemeld.pem | awk 'split_after==1{n++;split_after=0} /---- -END CERTIFICATE-----/ {split_after=1} {print > "minemeld" n ".cer"}' [minemeld ~]$ sudo cp minemeld.cer /etc/nginx/minemeld.cer [minemeld ~]$ sudo openssl rsa -in minemeld1.cer -out -out /etc/ngnix/minemeld/minemeld.pem [minemeld ~]$ sudo service nginx restart Now restart the browser session and you should not receive an untrusted error, if the root CA that signed the certificate is installed correctly on your machine.
View full article
Angelo ‎12-06-2016 06:53 PM
18,939 Views
7 Replies
2 Likes
Think of a prototype as a node template that can be instantiated inside the MineMeld engine config to define a new node.   Learn about customizing prototypes in this article.
View full article
lmori ‎07-21-2016 05:51 PM
24,210 Views
18 Replies
1 Like
Ask Questions Get Answers Join the Live Community