Parameters for the output feeds

cancel
Showing results for 
Show  only  | Search instead for 
Did you mean: 
L7 Applicator
100% helpful (6/6)

Each output node based on class minemeld.ft.redis.RedisSet has associated a feed accessible via the MineMeld API. The URL of the feed is shown in the node view.

feed-sshot.png

 

Additional parameters

You can use additional parameters on the feed URL to change the output format or the entry returned from the feed. You can combine multiple parameters in the same URL.

Parameter Description Example
(none) default format, the list of indicators is retrieved

https://minemeld/feeds/feed1

 

Result

1.10.16.0-1.10.31.255
1.116.0.0-1.119.4.98
1.119.12.116-1.119.255.255
1.119.4.100-1.119.12.114
1.32.128.0-1.32.191.255
101.192.0.0-101.195.255.255
101.202.0.0-101.202.255.255
101.203.128.0-101.203.159.255
101.248.0.0-101.249.235.117
101.249.235.119-101.249.255.255
101.252.0.0-101.253.255.255
103.16.76.0-103.16.76.255
103.2.44.0-103.2.47.255
[...]
s=<N> s=<N> retrieves entries starting from entry number N.

https://minemeld/feeds/feed1?s=3

 

Result

1.119.4.100-1.119.12.114
1.32.128.0-1.32.191.255
101.192.0.0-101.195.255.255
101.202.0.0-101.202.255.255
101.203.128.0-101.203.159.255
101.248.0.0-101.249.235.117
101.249.235.119-101.249.255.255
101.252.0.0-101.253.255.255
103.16.76.0-103.16.76.255
103.2.44.0-103.2.47.255
[...]
n=<M> n=<M> retrieves M entries from the feed. Can be combined with parameter s to select a subsect of the feed.

https://minemeld/feeds/feed1?s=3&n=2

 

Result

1.119.4.100-1.119.12.114
1.32.128.0-1.32.191.255
tr=1 translate IP ranges into CIDRs. This can be used also with v=json and v=csv.

https://minemeld/feeds/feed1?tr=1

 

Result

1.10.16.0/20
1.116.0.0/15
1.118.0.0/16
1.119.0.0/22
1.119.4.0/26
1.119.4.64/27
1.119.4.96/31
[...]
v=json

returns the indicator list in JSON format.

 

Note that the value of the indicator is returned only if the value flag is set in the prototype.

https://minemeld/feeds/feed1?v=json

 

Result

[
{"indicator":"1.10.16.0-1.10.31.255","value":{[...]
v=json-seq

returns the indicator list in JSON-SEQ format.

 

Note that the value of the indicator is returned only if the value flag is set in the prototype.

https://minemeld/feeds/feed1?v=json-seq

v=panosurl

if the feed contains URL indicators, they are returned in a format compatible with PAN-OS URL EDLs.

Optional attributes:

  • di=<anything> Drop Invalid entries. If an URL entry is not compliant with PAN-OS EDL URL format the entry is dropped instead of being rewritten
  • sp=<anythin> Strip Port. Ignores URL entries with ports instead of rewriting them

https://minemeld/feeds/feed1?v=panosurl

v=mwg returns the indicator list in a McAfee Web Gateway compatible format as described in https://community.mcafee.com/docs/DOC-5208

https://minemeld/feeds/feed1?v=mwg

 

Result

type=string
"iuqerfsodp9ifjaposdfjhgosurijfaewrwergwea.com" "WanaCrypt0r_Miner"
"www.iuqerfsodp9ifjaposdfjhgosurijfaewrwergwea.com" "WanaCrypt0r_Miner"
"iuyuip.com" "WanaCrypt0r_Miner"
"oaagmx.com" "WanaCrypt0r_Miner"

In the case the indicator feed is composed by IP addresses then you can modify the output type with the t=ip additional attribute

 

Example:

https://minemeld/feeds/feed1?v=mwg&t=ip

 

Result

type=ip
"82.195.75.101" "WanaCrypt0r_Miner"
"1.211.23.1" "WanaCrypt0r_Miner"
"1.211.23.152" "WanaCrypt0r_Miner"
"1.211.23.2" "WanaCrypt0r_Miner"
"101.159.183.1" "WanaCrypt0r_Miner"
"101.52.197.161" "WanaCrypt0r_Miner"
"102.224.162.252" "WanaCrypt0r_Miner"
"11.175.27.1" "WanaCrypt0r_Miner"
v=bluecoat returns the indicator list in a BlueCoat Local List format as described in this Technical Brief document

Optional attributes:

  • cd=<category_name> (Category Default): Default Category where the indicators will be placed to
  • ca=<attribute_name> (Category Attribute): The indicator might have an additional attribute with a list of strings describing the categories it should be listed on.

Example:

https://minemeld/feeds/feed1?v=bluecoat&cd=FROMAUTOFOCUS&ca=bc_category

 

Result

define category MM_MALWARE
phishyou.foobar.com
end
define category FROMAUTOFOCUS
iuqerfsodp9ifjaposdfjhgosurijfaewrwergwea.com
www.iuqerfsodp9ifjaposdfjhgosurijfaewrwergwea.com
iuyuip.com
oaagmx.com
end
v=csv

returns the indicator list in CSV format.

 

The list of the attributes is specified by using the parameter f one or more times. The default name of the column is the name of the attribute, to specify a column name add |column_name in the f parameter value.

 

The h parameter can be used to control the generation of the CSV header. When unset (h=0) the header is not generated. Default: set.

 

Encoding is utf-8. By default no UTF-8 BOM is generated. If ubom=1 is added to the parameter list, a UTF-8 BOM is generated for compatibility.

 

https://minemeld/feeds/feed1?v=csv&f=confidence&f=sources|feeds&f=indicator|clientip&f=dshield_email

 

confidence,feeds,clientip,dshield_email
100,dshield.block,104.193.252.0/24,abuse@king-servers.com
...

 

Rate this article:
Comments
L2 Linker

Hello -

 

I have created an EDL in PANOS 8.0.0 using a feed from Minemeld 0.9.40, when I commit I receive the following message:

 

EDL(vsys1/Skype-IPv4 ip) Downloaded file is not a text file.

 

Does anyone know how to correct the error ?

 

Thanks

L7 Applicator

Hi @paul_w,

could you open discussion under MineMeld discussions about this issue ? 99% probability this is a connectivity issue or certificate issue, I know the PAN-OS error message is misleading.

L0 Member

When I am trying to download feeds using Curl script and below API URL, only IP address information is getting, not confidence value and sources detail.

 

https://minemeld/feeds/feed1?tr=1&v=csv&f=indicator|clientip&f=confidence&f=sources|feeds

 

Does anyone know how to fix the issue?

L5 Sessionator

@MohammedS,

 

you must be working on a output node whose prototype do not enable the storage of "values" (metadata of the indicator).

 

If you're using nodes from the standard library then chose the ones with the "WithValue" suffix in the name.2018-02-16_07-49-16.png

 

If you're creating your own prototypes then make sure you enable the "store_value" configuration attribute.

 

2018-02-16_07-49-53.png

 

 

 

 

 

 

L1 Bithead

@lmori I see that a couple of additional output formats have been added. Is it possible to create an output format for Bro/Zeek Intel Framework? The CIDRs output format gets close but Bro doesn't seem to be able to accept anything except individual IP addresses so the output would have to break out a /24 into 256 individual IPs and etc. for other CIDRs in the output. Thanks in advance!

L0 Member

I have several miner nodes reporting into 5 processors (FQDN,URL,IPV4,etc.).  When I create the output node I'm limited to chose a single processor.  Is there a way to configure MM to use multiple processor nodes.  I found the configuration on a higher ed article on the REN-ISAC site and it directed to create the separate nodes.   

 

As far as the URL and FQDNs feeds do they have to have dedicated output node?  I'm getting 153k IOCs and can only transfer 32,000 to my Palo Alto.  Any info on what I need to do would be appreciated. 

  • 42832 Views
  • 6 comments
  • 7 Likes
Register or Sign-in
Contributors
Labels
Article Dashboard
Version history
Last Updated:
‎12-02-2019 02:58 AM
Updated by: