11-07-2016 05:39 AM
Maybe a stupid question and\or I've missed the obvious...
One of the issues we have with our Palo firewalls is - when we deploy 'active' IPS rules (block-ip etc) the maximum length of time is 3600 seconds.
We have a log solution that we use to trigger alerts if we're being probed over multiple days etc and would like to trigger a script, rather than the current manual email, to poke the offending IP address into a block list. As we've already started using MineMeld to serve up blocklists as I wondering if its possible to call a RESTful API - or similar - to push the IP address to an output list?
Longer term we'd be looking for a bit more intelligence i.e. checking whether it's already on the list, removing after a period of time (say 7 days), but initially it would just be a simple "if source IP triggers threat 3 times in 3 hours trigger (PowerShell) script to poke address into custom blocklist" type scenario
11-07-2016 11:50 PM
Hi @apackard,
MineMeld has a REST API you can use to add indicators to lists. You could use that to push block-ips.
Is there a public documentation of your log solution I could use to check their integration capabilities ?
Thanks,
luigi
11-08-2016 06:44 AM
Hey @lmori,
Do you have documentation anywhere for using the Minemeld API? I have been hunting live and the github wiki to no avail. (Nice docs btw!) Sorry if you've already posted this somewhere.
I'm trying to help my customer automate some integration between ServiceNow and PAN for their approval process for whitelisting domains and IPs, and it would be great to use Minemeld EDLs/DAGs to eliminate the need to commit.
I currently have a python script that goes and manually updates the indicator YML files via paramiko, but I'd prefer to use the API.
Thanks again for the great tool and looking forward to continued improvement.
-Nasir
11-08-2016 07:40 AM - edited 11-08-2016 07:41 AM
This is the basic way to add an indicator to a list.
Scenario: you have a Miner of type stdlib.listIPv4 with name my_ipv4_list
To add 1.1.1.2 to the list of indicators:
curl -XPOST -H "Content-Type: application/json" -u <username>:<password> https://<minemeld>/config/data/my_ipv4_list_indicators/append?h=my_ipv4_list -d '{
"indicator": "1.1.1.2",
"type": "IPv4",
"share_level": "red",
"comment": "Test"
}'
(do not forget the Content-Type header)
11-09-2016 09:22 AM - edited 11-10-2016 06:56 AM
Thanks @lmori, this is gold! How about to remove an indicator?
Would I just replace the "append" keyword with another method? I've tried "delete", "del", "remove", "rem", "erase", "strip", and other similar verbs to no avail. It'd be awesome if we could put together a real simple document outlining use of the API.
BTW here's a working sample of Python code we're building into our API integration for adding indicators. It'll be very handy if we can get a list of supported actions with their syntax so that we can modify and remove indicators as well.
import pycurl, json
def add_indicator_api(**kwargs):
url = "https://{hostname}:{port}/config/data/{miner}_indicators/append?h={miner}".format(**kwargs)
payload = json.dumps({"indicator": "{indicator}".format(**kwargs), "share_level": "green", "comment": "{comment}".format(**kwargs)})
curl = pycurl.Curl()
curl.setopt(pycurl.SSL_VERIFYHOST, 0)
curl.setopt(pycurl.SSL_VERIFYPEER, 0)
curl.setopt(pycurl.URL, url)
curl.setopt(pycurl.POST, 1)
curl.setopt(pycurl.HTTPHEADER, ['Content-Type: application/json'])
curl.setopt(pycurl.HTTPAUTH, pycurl.HTTPAUTH_BASIC)
curl.setopt(pycurl.USERPWD, "{username}:{password}".format(**kwargs))
curl.setopt(pycurl.POSTFIELDS, payload)
curl.perform()
curl.close()
11-10-2016 01:33 PM
Hi @nbilal,
currently you can only append, get the full list or post a full list. I am planning to add a remove action in the next release.
See the last lines of this Python file here for the API:
https://github.com/PaloAltoNetworks/minemeld-core/blob/master/minemeld/flask/configapi.py#L568
For python I would suggest to requests to build an API client.
11-10-2016 09:05 PM - edited 11-10-2016 09:06 PM
Great info @lmori!
I rewrote my method to use requests:
import requests, json
# PYTHON REQUESTS API CLIENT
def add_indicator_requests(**kwargs):
url = "https://{hostname}:{port}/config/data/{miner}_indicators/append?h={miner}".format(**kwargs)
payload = json.dumps({
"indicator": "{indicator}".format(**kwargs),
"share_level": "green",
"comment": "{comment}".format(**kwargs)
})
headers = {'Content-Type': 'application/json'}
requests.post(
url, data=payload,
headers=headers,
auth=("{username}".format(**kwargs), "{password}".format(**kwargs)),
verify=False
)
I also wrote a simple "delete" method (GET whole indicator file, remove specified JSON, overwrite original YAML with modified JSON data):
# GET > MODIFY DATA (save as jlist) > PUT
def delete_indicator_requests(**kwargs):
url = "https://{hostname}:{port}/config/data/{miner}_indicators".format(**kwargs)
headers = {'Content-Type': 'application/json'}
req = requests.get(
url,
auth=("{username}".format(**kwargs), "{password}".format(**kwargs)),
verify=False
)
jlist = []
for j in req.json()['result']:
if j['indicator'] != "{indicator}".format(**kwargs):
jlist.append(j)
requests.put(
url, json=jlist,
headers=headers,
auth=("{username}".format(**kwargs), "{password}".format(**kwargs)),
verify=False
)
Thanks again! We've finally got this thing doing our bidding.
11-14-2016 02:39 AM
Hi @nbilal,
that's what the UI does when you delete an indicator. Not super efficient but it works. We have plans to improve these Miners or add new one able to handle large numbers of Indicators.
12-29-2016 12:59 PM
For those that are working in MS Windows shops here is something i worked up in powershell today. Please forgive me because I am not a coder so some logic could probably be improved and I am not complete with all error checking.
My original use case was to pull a list of all our Domain Controller IP's and add them to a EDL, for some obvious reasons to those on this this. Powershell made the most sense and I began going down that rabbit hole.
First was a powershell script to retrieve the DC:
Get-ADDomainController -Filter * | foreach {
if ($ip.Length -gt 7)
{
#Some block of code to add the IP address
addIndicator($ip)
}
}
That was simple enough ... The next part was a little more complicated but non the less here is the function that actually adds the IP addresses:
$mineMeldHost = "https://{minemeldServer-FQDN-or-IP}"
#indicatorOutput is the output/feed list name that the indicators should exist in
$indicatorOutput = "Domain_Controller_List"
$url = $mineMeldHost + "/feeds/" + $indicatorOutput
$currentList = Invoke-WebRequest $url
Function addIndicator ($indicator)
{
#Verify the IP address is not already included in the list
if ( -not $currentList.Content.Contains($indicator) )
{
$IndicatorList = "Domain_Controllers"
#Credentials can be passed using basic authentication
# * Simply base46 encode {username}:{password} and add that string to the headers
# * Be sure to include the ':' between the strings
# * $userPass = [Convert]::ToBase64String([Text.Encoding]::ASCII.GetBytes("$($user):$($passwd)"))
$userPass = "{Base64String}"
# Adding the Authentication string to the post request headers
$headers = @{
Authorization = 'Basic ' + $userPass
}
# to add date and time to the comments field for each indicator
$dateTime = get-date -format F
$comment = 'Indicator added with Powershell on ' + $dateTime
$indicatorArr = @{
indicator = $indicator
type = 'IPv4'
share_level = 'red'
comment = $comment
}
$json = $indicatorArr | ConvertTo-Json
$url = $mineMeldHost + "/config/data/" + $IndicatorList + "_indicators/append?h=" + $IndicatorList
$response = Invoke-RestMethod $url -Method Post -Body $json -ContentType 'application/json' -Headers $headers
$response
Write-Output "The Following Indicator was added: $indicator"
}
}
I have since seen some potential to include this is some Incident Response Scenarios so started working on cleaning the code up and giving it some switches and options.
There are a few defaults that need to be changed, and the base64encoded username:password needs to be included .
<#
Author: Sean Engelbrecht
Version: 2016.12.29.0.1
Version History
Version: 2016.12.29.0.1
Original Code
#>
Function Add-Indicator
<#
.Synopsis
Add indicators to MineMeld feeds utilized by Palo Alto Firewalls
.DESCRIPTION
This cmdlet can be utilized to add threat indicators to fields listed in minemeld ( A Palo Alto open source threat aggregation tool).
Mandatory functions for this function include; Server, FeedList, IndicatorList, Type and Indicator.
.PARRAMETER Server
This Parameter contains the ip-address or FQDN of the MineMeld server.
Parameter has no Default Value
.PARAMETER Indicator
This Parameter contains the Indicator to be added to the MineMeld server.
Parameter has no Default Value
.PARAMETER Type
This Parameter contains the type of indicator to be added to the the MineMeld server (IPv4 or URL).
Parameter Default Value: URL
.PARAMETER IndicatorList
This Parameter contains the name of the input stream/list where the indicator should be added.
Parameter Default Value: {name-of-your-input-stream}
.PARAMETER FeedList
This Parameter contains the name of the output stream/list where the indicator should be added.
Parameter Default Value: {name-of-your-output-stream}
.PARAMETER IncludeSubDomain
If this parameter is present and the Type is URL an additional indicator will be added containing a wildcard token.
.PARAMETER BypassSSLError
If this parameter is present self-signed certificate errors will be bypassed.
.PARAMETER ShareLevel
This Parameter contains the share level of indicator to be added to the the MineMeld server.
Parameter Default Value: red
.EXAMPLE
Add-Indicator -Server 192.168.1.10 -Indicator "evil.com"
This will add the url evil.com to the default list on minemeld server (192.168.1.1)
.EXAMPLE
Add-Indicator -Server 192.168.1.10 -Indicator "evil.com" -IncludeSubDomain
Will add the url's evil.com and *.evil.com to the default list on minemeld server (192.168.1.1)
.EXAMPLE
Add-Indicator -Server 192.168.1.10 -Indicator "evil.com" -BypassSSLError
This will add the url evil.com to the default list on minemeld server (192.168.1.1) and bypass and SSL certificate errors caused by self-signed SSL certs.
.EXAMPLE
Add-Indicator -Server 192.168.1.10 -Indicator "172.16.12.21" -Type IPv4 -FeedList "mm_dc_list" -IndicatorList "DC_IP_List"
Will Add ip address 172.16.21.21 to mm_dc_list on minemeld server (192.168.1.1)
#>
{
[CmdletBinding()]
Param(
[parameter(Mandatory=$true,
valueFromPipelineByPropertyName=$true,
HelpMessage="IP-Address or FQDN of MineMeld Server:",
Position=0)]
[String[]]
$Server,
[parameter(Mandatory=$false,
valueFromPipelineByPropertyName=$true,
HelpMessage="Output stream of threat feed:",
Position=3)]
[String[]]
$FeedList = "{name-of-your-output-stream}",
[parameter(Mandatory=$false,
valueFromPipelineByPropertyName=$true,
HelpMessage="Input stream to add threat indicator to:",
Position=4)]
[String[]]
$IndicatorList = "{name-of-your-input-stream}",
[parameter(Mandatory=$false,
valueFromPipelineByPropertyName=$true,
HelpMessage="Indicator type[IPv4 or URL]:",
Position=1)]
[string]
[validateSet("IPv4","URL")]
$Type = "URL",
[parameter(Mandatory=$false,
valueFromPipelineByPropertyName=$true,
HelpMessage="Indicator type[IPv4 or URL]:",
Position=5)]
[string]
[validateSet("green","yellow","red")]
$ShareLevel = "red",
[parameter(Mandatory=$true,
valueFromPipelineByPropertyName=$true,
HelpMessage="Threat indicator to Add:",
Position=2)]
[String[]]
$Indicator,
[parameter(Mandatory=$false,
valueFromPipelineByPropertyName=$true,
HelpMessage="Bypass any SSL Errors:",
Position=7)]
[switch]
$BypassSSLError,
[parameter(Mandatory=$false,
valueFromPipelineByPropertyName=$true,
HelpMessage="Include wildcard character to for subdomains *.evil.com:",
Position=6)]
[switch]
$IncludeSubDomain
)
Begin
{
#Set Initial Variables and check for errors
If ($BypassSSLError)
{
if (-not ([System.Management.Automation.PSTypeName]'TrustAllCertsPolicy').Type)
{
add-type @"
using System.Net;
using System.Security.Cryptography.X509Certificates;
public class TrustAllCertsPolicy : ICertificatePolicy {
public bool CheckValidationResult(
ServicePoint srvPoint, X509Certificate certificate,
WebRequest request, int certificateProblem) {
return true;
}
}
"@
[System.Net.ServicePointManager]::CertificatePolicy = New-Object TrustAllCertsPolicy
}
}
$stage = 0
# Variable used to the while loop needed to include a wildcard for subdomains
$exitLoop = $false
}
Process
{
$url = "https://" + $Server + "/feeds/" + $FeedList
Try
{
$stage = 1
# Retrieve the current feed list, this prevents duplicate entries.
$currentList = Invoke-WebRequest $url -TimeoutSec 30
$stage = 2
# Check if indicator exists
if ( -not $currentList.Content.Contains($Indicator) )
{
#Credentials can be passed using basic authentication
# * Simply base46 encode {username}:{password} and add that string to the headers
# * Be sure to include the ':' between the strings
# * $userPass = [Convert]::ToBase64String([Text.Encoding]::ASCII.GetBytes("$($user):$($passwd)"))
$userPass = "{base64-of-your-username:password}"
# Adding the Authentication string to the post request headers
$Headers = @{
Authorization = 'Basic ' + $userPass
}
while ( -not $exitLoop)
{
# to add date and time to the comments field for each indicator
$dateTime = get-date -format F
$Comment = 'Indicator added with Powershell on ' + $dateTime
# Array that will be converted to JSON format for POST request
$indicatorArr = @{
indicator = "$Indicator"
type = "$Type"
share_level = "$ShareLevel"
comment = "$Comment"
}
$requestBody = $indicatorArr | ConvertTo-Json
$url = "https://" + $Server + "/config/data/" + $IndicatorList + "_indicators/append?h=" + $IndicatorList
$Response = Invoke-RestMethod $url -Method Post -Body $requestBody -ContentType 'application/json' -Headers $Headers
Write-Output "The Following Indicator was added: $indicator"
if ( "$Type" -eq "URL" )
{
# Process structure for URL Indicators
if ( ($IncludeSubDomain -and $Indicator.Get(0).Contains("*.") ) -or -not $IncludeSubDomain )
{
# If the wildcard has been processed already, or there is no need to include sub-domains, exit the loop.
$exitLoop = $true
}
else
# Since sub-domains are to be included, loop back around and add additional indicator with wildcard token.
{
$Indicator = "*.$Indicator"
}
}
else
# If the Indicator is an IPv4 type, simply exit the loop.
{
$exitLoop = $true
}
}
}
}
catch
{
switch ($stage)
{
1 { throw "Error retrieving data from '$url', Indicators cannot be added at this time." }
2 { throw "Error adding Indicator, ($Indicator) to $IndicatorList on $Server." }
default {"Unknown Error..."}
}
}
}
end
{
#Print function status and cleanup
Write-Host "Done"
}
}
Feel free to use, update and modify as needed.
--Sean
02-02-2017 09:06 AM
Updated to fix silly mistakes:
<#
Author: Sean Engelbrecht
Version: 2017.02.02.0.3
Version History
Version: 2017.02.02.0.3
Fix: Corrected logic flow on verifying the indicator
Enhance: Error output will alert if ssl error was reason for failure.
Version: 2017.01.04.0.2
Fix: Help Content
Get-Help Add-Indicator -Full would not properly display information
Fix: Variable Types
Some variables were set to String Arrays, corrected not all string variables are type string
Version: 2016.12.28.0.1
Original Code
#>
Function Add-Indicator
{
<#
.SYNOPSIS
Add indicators to MineMeld feeds utilized by Palo Alto Firewalls
.DESCRIPTION
This cmdlet can be utilized to add threat indicators to fields listed in minemeld ( A Palo Alto open source threat aggregation tool).
Mandatory functions for this function include; Server, FeedList, IndicatorList, Type and Indicator.
.PARAMETER Server
This Parameter contains the ip-address or FQDN of the MineMeld server.
Parameter has no Default Value
.PARAMETER Indicator
This Parameter contains the Indicator to be added to the MineMeld server.
Parameter has no Default Value
.PARAMETER Type
This Parameter contains the type of indicator to be added to the the MineMeld server (IPv4 or URL).
Parameter Default Value: URL
.PARAMETER IndicatorList
This Parameter contains the name of the input stream/list where the indicator should be added.
Parameter Default Value: dvn_Malware_List
.PARAMETER FeedList
This Parameter contains the name of the output stream/list where the indicator should be added.
Parameter Default Value: HC_URL_List
.PARAMETER IncludeSubDomain
If this parameter is present and the Type is URL an additional indicator will be added containing a wildcard token.
.PARAMETER BypassSSLError
If this parameter is present self-signed certificate errors will be bypassed.
.EXAMPLE
Add-Indicator -Server 192.168.1.10 -Indicator "evil.com"
This will add the url evil.com to the default list on minemeld server (192.168.1.1)
.EXAMPLE
Add-Indicator -Server 192.168.1.10 -Indicator "evil.com" -IncludeSubDomain
Will add the url's evil.com and *.evil.com to the default list on minemeld server (192.168.1.1)
.EXAMPLE
Add-Indicator -Server 192.168.1.10 -Indicator "evil.com" -BypassSSLError
This will add the url evil.com to the default list on minemeld server (192.168.1.1) and bypass and SSL certificate errors caused by self-signed SSL certs.
.EXAMPLE
Add-Indicator -Server 192.168.1.10 -Indicator "172.16.12.21" -Type IPv4 -FeedList "mm_dc_list" -IndicatorList "DC_IP_List"
Will Add ip address 172.16.21.21 to mm_dc_list on minemeld server (192.168.1.1)
#>
[CmdletBinding()]
Param(
[parameter(Mandatory=$true,
valueFromPipelineByPropertyName=$true,
HelpMessage="IP-Address or FQDN of MineMeld Server:",
Position=0)]
[String]
$Server,
[parameter(Mandatory=$false,
valueFromPipelineByPropertyName=$true,
HelpMessage="Output stream of threat feed:",
Position=3)]
[String]
$FeedList = "HC_URL_List",
[parameter(Mandatory=$false,
valueFromPipelineByPropertyName=$true,
HelpMessage="Input stream to add threat indicator to:",
Position=4)]
[String]
$IndicatorList = "dvn_Malware_List",
[parameter(Mandatory=$false,
valueFromPipelineByPropertyName=$true,
HelpMessage="Indicator type[IPv4 or URL]:",
Position=1)]
[string]
[validateSet("IPv4","URL")]
$Type = "URL",
[parameter(Mandatory=$false,
valueFromPipelineByPropertyName=$true,
HelpMessage="Indicator type[IPv4 or URL]:",
Position=5)]
[string]
[validateSet("green","yellow","red")]
$ShareLevel = "red",
[parameter(Mandatory=$true,
valueFromPipelineByPropertyName=$true,
HelpMessage="Threat indicator to Add:",
Position=2)]
[String]
$Indicator,
[parameter(Mandatory=$false,
valueFromPipelineByPropertyName=$true,
HelpMessage="Bypass any SSL Errors:",
Position=7)]
[switch]
$BypassSSLError,
[parameter(Mandatory=$false,
valueFromPipelineByPropertyName=$true,
HelpMessage="Include wildcard character to for subdomains *.evil.com:",
Position=6)]
[switch]
$IncludeSubDomain
)
Begin
{
#Set Initial Variables and check for errors
If ($BypassSSLError)
{
#if (-not ([System.Management.Automation.PSTypeName]'TrustAllCertsPolicy').Type)
#{
add-type @"
using System.Net;
using System.Security.Cryptography.X509Certificates;
public class TrustAllCertsPolicy : ICertificatePolicy {
public bool CheckValidationResult(
ServicePoint srvPoint, X509Certificate certificate,
WebRequest request, int certificateProblem) {
return true;
}
}
"@
[System.Net.ServicePointManager]::CertificatePolicy = New-Object TrustAllCertsPolicy
#}
}
# Variabel used to the while loop needed to include a wildcard for subdomains
$exitLoop = $false
}
Process
{
$url = "https://" + $Server + "/feeds/" + $FeedList
Try
{
$Error.Clear()
$stage = 1
# Retrieve the current feed list, this prevents duplicate entries.
$currentList = Invoke-WebRequest $url -TimeoutSec 30
$stage = 2
#Credentials can be passed using basic authentication
# * Simply base46 encode {username}:{password} and add that string to the headers
# * Be sure to include the ':' between the strings
$userPass = [Convert]::ToBase64String([Text.Encoding]::ASCII.GetBytes("$($user):$($passwd)"))
# Adding the Authentication string to the post request headers
$Headers = @{
Authorization = 'Basic ' + $userPass
}
while ( -not $exitLoop)
{
# Check if indicator exists
if ( -not $currentList.Content.Contains($Indicator) )
{
# to add date and time to the comments field for each indicator
$dateTime = get-date -format F
$Comment = 'Indicator added with Powershell on ' + $dateTime
# Array that will be converted to JSON format for POST request
$indicatorArr = @{
indicator = "$Indicator"
type = "$Type"
share_level = "$ShareLevel"
comment = "$Comment"
}
$requestBody = $indicatorArr | ConvertTo-Json
$url = "https://" + $Server + "/config/data/" + $IndicatorList + "_indicators/append?h=" + $IndicatorList
$Response = Invoke-RestMethod $url -Method Post -Body $requestBody -ContentType 'application/json' -Headers $Headers
Write-out "The Following Indicator was added: $indicator"
}
else
{
Write-out "The Following Indicator was skipped, already in the list: $indicator"
}
if ( "$Type" -eq "URL" )
{
# Process structure for URL Indicators
if ( ($IncludeSubDomain -and $Indicator.Contains("*.") ) -or -not $IncludeSubDomain )
{
# If the wildcard has been processed already, or there is no need to include sub-domains, exit the loop.
$exitLoop = $true
}
else
# Since sub-domains are to be included, loop back around and add additional indicator with wildcard token.
{
$Indicator = "*.$Indicator"
$stage = 3
}
}
else
# If the Indicator is an IPv4 type, simply exit the loop.
{
$exitLoop = $true
}
}
}
catch
{
switch ($stage)
{
1 {
if ( $Error.Item(0).toString().Contains("Could not establish trust relationship for the SSL" ))
{
throw "Error: Could not establish trust relationship for the SSL/TLS secure channel for '$url', Indicators cannot be added at this time."
}
else
{
throw "Error retrieving data from '$url', Indicators cannot be added at this time."
}
}
2 { throw "Error adding Indicator, ($Indicator) to $IndicatorList on $Server." }
3 { throw "Error adding Wildcard Indicator, ($Indicator) to $IndicatorList on $Server." }
default {"Unknown Error... Thanks Obama!"}
}
}
}
end
{
#Print function status and cleanup
Write-Verbose "Done"
}
}
01-26-2018 12:49 AM - edited 01-26-2018 12:50 AM
Sean, nice script! I took your script a step further and incorporated a GUI along with other logic to allow for easy uploading of IOC's. I made sure to cite your contribution:
04-30-2018 03:27 PM
Did you add any more methods (like delete)?
05-04-2018 01:33 AM
Hi @pataylor,
delete is available in the new API, you can check the new API mechanism at the end of the following article:
Click Accept as Solution to acknowledge that the answer to your question has been provided.
The button appears next to the replies on topics you’ve started. The member who gave the solution and all future visitors to this topic will appreciate it!
These simple actions take just seconds of your time, but go a long way in showing appreciation for community members and the LIVEcommunity as a whole!
The LIVEcommunity thanks you for your participation!
