Customizing Prototypes

by lmori on ‎02-03-2016 01:59 AM - edited on ‎07-21-2016 05:51 PM by Community Manager (6,937 Views)

As explained in this article, you can think of a prototype as a node template that can be instantiated inside the MineMeld engine config to define a new node. 

 

Prototype Libraries

In the default MineMeld installation, prototypes are stored inside prototypes libraries located in 2 different directories:

- /opt/minemeld/prototypes/current contains the standard prototypes libraries. These are automatically updated by the MineMeld auto update mechanism.

- /opt/minemeld/local/prototypes contains local prototypes libraries. This is the directory you want to put your own prototypes. 

 

A prototype library is a YAML file with the following structure (from dshield.yml library):

# library author, optional
author: lmori
# URL with more details, optional
url: https://www.dshield.org/xml.html
# library description, recommended
description: >
The ISC uses the DShield distributed intrusion detection system for
data collection and analysis. DShield collects data about malicious
activity from across the Internet. This data is cataloged and summarized
and can be used to discover trends in activity, confirm widespread
attacks, or assist in preparing better firewall rules.

# list of prototypes, mandatory
prototypes:
# prototype name, mandatory. Should be unique inside the library
block:
# development status, recommended
development_status: STABLE
# node type, recommended
node_type: miner
# description of the prototype, recommended
description: suggested block list
# node class, mandatory
class: minemeld.ft.http.HttpFT
# node config, recommended
config:
[...]

 

Inside the MineMeld Engine Config

Inside the MineMeld engine config file prototypes are used inside the node definition. You can check the running config file /opt/minemeld/local/config/running-config.yml for an example:

nodes:
# ... more nodes here
dshield_blocklist:
output: true
prototype: dshield.block
# ... nore nodes here

 

A prototype is referenced as <library name>.<prototype name>.

 

Customizing a Prototype

The easiest way to customize a prototype is creating your own local version of the prototype and then use it inside the config.

 

As an example we will create a local version of the malwaredomainlist.ip prototype to raise the confidence of the indicators.

 

1. Copy the library to the local prototype directory

Copy the original library to a new library with a new unique global name in the local prototype directory

$ sudo -u minemeld cp /opt/minemeld/prototypes/current/malwaredomainlist.yml /opt/minemeld/local/prototypes/myownmalwaredomainlist.yml

 

2. Edit the new library

And override the confidence attribute (and optionally description):

author: foobar
description: Local version of Malware Domain List library.
url: https://www.malwaredomainlist.com/

prototypes:
ip:
development_status: EXPERIMENTAL
node_type: miner
description: list of active ip addresses
class: minemeld.ft.http.HttpFT
config:
source_name: malwaredomainlist.ip
attributes:
type: IPv4
direction: outbound
share_level: green
confidence: 75
url: http://www.malwaredomainlist.com/hostslist/ip.txt

 

3. Use the new prototype inside the UI

You may need to reload the UI in the browser to load the new prototype, there is no need to restart the engine or the web frontend.

Comments
by netmaster_UC3M
on ‎06-13-2016 05:47 AM

Hello,

 

We are trying to integrate badips.com lists, but the recipe on this page does not work for us. Here is our UC3M.yml file.

We had restarted mimemeld and even the system. But no luck, it should be a simple mistake, but we are unable to find it.

 

Could anyone help us?

 

 

author: certuc3m
description: >
       A comunity driven repository
url: https://www.badips.com

prototypes:
CommunityBadIPs:
development_status: STABLE
node_type: miner
description: badips.com is a community based IP blacklist service.
class: minemeld.ft.http.HttpFT
config:
source_name: UC3M.2w
url: https://www.badips.com/get/list/any/3?age=2w
attributes:
type: IPv4
direction: inbound
share_level: green
confidence: 50


 

by lmori
on ‎06-13-2016 06:25 AM

Hi netmaster_UC3M,

please, could you open a discussion with your prototype under MineMeld discussions ? I'll be happy to help.

by Kimwii
on ‎01-30-2017 01:53 AM

I'm currently in the process of creating a new prototype to pull a feed from a trusted source.

The feed uses basicauth so i tried to do as following url: https://username:password@domain.tld

But no luck.

Is there a trick i need to know about when it comes to password protected feeds with BasicAuth or other types of authentication ?

by lmori
on ‎01-30-2017 06:42 AM

Hi @Kimwii,

currently (0.9.30) plain Http Miner does not support Basic Auth, you would need to extend it to support Basic Auth.

by hsoni
on ‎02-13-2017 06:43 AM

Now minemeld is integrated with Autofocus. How can we update custom prototype from UI?

 

Is there any option in UI to make changes to existing prototype?

by lmori
on ‎02-15-2017 01:33 PM

Hi @hsolis,

yes, just click on an existing prototype and then click NEW in the top right corner. This will create a copy of the current prototype you can customize and save.

by SantiBT
on ‎02-20-2017 09:01 AM

Hi,

 

Do you know something sample about integration with MISP (Malware Information share platform)???

 

So another question is about scripts, can I launch a script into conifg a new prototype? If I've created a new prototype I set a url option...can I set the url option for script option????

 

Thanks a lot

by Kimwii
on ‎02-20-2017 11:04 PM

I think misp has it's own feed tools.
But if you want to integrate the feed into misp i recommend using pymisp.

https://github.com/CIRCL/PyMISP

https://media.readthedocs.org/pdf/pymisp/master/pymisp.pdf

 

USE the api call below 

add_url(event, url, category=’Network activity’, to_ids=True, comment=None, distribution=None, proposal=False)

 

Remembner to change your key file and add your api key for misp

 

by SantiBT
on ‎02-21-2017 04:09 AM

Hi Kimwii

 

Do you refer about file config??? such us:

 

attributes:
confidence: 80
share_level: green
type: IPv4
delimiter: '#'
fieldnames:
- indicator
interval: 3600
source_name: misp.reputation.ip-dst
url: http://ip_myMisp/attributes/text/download/ip-dst
add_url(event, url, category=’Network activity’, to_ids=True, comment=None, distribution=None, proposal=False)

 

minemeld.PNG

 

 

is it so??????

 

Thx

by SantiBT
on ‎02-21-2017 04:11 AM

So I want to integrate MISP into Minemeld, I don't want to integrate Minemeld into MISP.

 

How I can integrate MISP into Minemeld??????

 

Regards!

by Kimwii
on ‎02-28-2017 03:21 AM

@lmori hi again.
We talked about feeds an basicauth some timeago and i wondered how do i get minemeld to "support" basic auth feeds ?.

Is there som magic trick or is it an rfc

by lmori
on ‎02-28-2017 07:52 AM

Hi @Kimwii,

this should be addressed and supported in the next release.

by Kimwii
on ‎04-18-2017 11:12 AM

@lmori still having hazzle with creating a json prototype with basic auth for this format

 

{
  "something1": [
    "somedomain1.tld",
    "somedomain2.tld"
  ],
  "something2": [
    "somedomain3.tld",
    "somedomain4.tld"
  ],
  "something3": [
    "somedomain5.tld",
 "somedomain6.tld",
  ]
}

 

Any tips or ticks ? :).

 

Also  want to recomend minemeld anyone who'd like a easy way to manage their IOCs.

@lmori is super awsome and always helpfull.

 

Also is there a place i can write a review of minemeld here at paloaltos website ?.
That's the least i can do for everything you guys have done to make my day easier

by lmori
‎04-19-2017 02:05 AM - edited ‎04-19-2017 02:08 AM

HI @Kimwii,

you should create a prototype based on minemeld.ft.json.SimpleJSON (you can start from aws.AMAZON prototype), and configure it like this (note that the following prototype won't work because S3 does not accept Basic Auth headers):

 

age_out:
    default: null
    sudden_death: true
attributes:
    confidence: 100
    share_level: green
    type: domain
extractor: '*[].{ indicator: @ }'
indicator: indicator
prefix: test
source_name: test.kim
url: https://s3-eu-west-1.amazonaws.com/minemeld-test/test.json
username: guest
password: guest

extractor and indicator are JMESPath expressions, extract should be used to extract a list of objects from the JSON document while indicator can be used to identify the indicator inside each object.

 

@Kimwii, I am really glad you like MineMeld. I would be happy to see an article from yours about all the cool things you are doing with Splunk & MineMeld :-)

by cong
3 weeks ago - last edited 3 weeks ago

I understand that Soltra is part of the existing 3rd party intelligence feed, just wondering has anyone created a prototype from FS-ISAC? THe portal address is https://portal.fsisac.com/

 

Understand from FS-ISAC, they uses Soltra as part of their intel too, is FS-ISAC intelligence pool as subset of Soltra?

by lmori
3 weeks ago

Hi @cong,

please, could you open a discussion under MineMeld Discussions about this ? Will be happy to help you.

by cong
3 weeks ago - last edited 2 weeks ago

Hi @lmori,

 

I had opened a discussion, please find the link https://live.paloaltonetworks.com/t5/MineMeld-Discussions/Prototype-for-FS-ISAC/m-p/159162#M1132, really appreciate your help. 

 

Regards

Learn more
Ask Questions Get Answers Join the Live Community
Labels
Contributors