topic Re: STIX and TAXII support in General Topics

STIX and TAXII support

ScottyAU — Wed, 11 May 2016 04:58:43 GMT

Hi all,

Anyone used minemeld with STIX and TAXII? While we pretty familure with STIX/TAXII - only just booted minemeld for the first time.

Cheers,

Scotty

Re: STIX and TAXII support

lmori — Wed, 11 May 2016 05:10:22 GMT

Hi Scotty,

yes, there are some MineMeld instances out there retrieving indicators via STIX/TAXII, from TIPs mainly.

There are a couple of prototypes for hailataxii feeds in the prototype library you can check as examples.

Let me know if you need some help in setting up the TAXII Miner.

Re: STIX and TAXII support

ScottyAU — Wed, 11 May 2016 05:12:51 GMT

Thanks for the quick response lmori.

I'll have a look at setting up a miner and see how i get on.

We do require both username/password as well as a client certificate for our taxii server - any idea if that is supported?

Cheers!

Scotty

Re: STIX and TAXII support

lmori — Wed, 11 May 2016 05:20:39 GMT

Hi Scotty,

usernamd and password are supported, client certificate not yet. Is it mandatory for your TAXII server ?

Thanks,

luigi

Re: STIX and TAXII support

ScottyAU — Wed, 11 May 2016 05:23:43 GMT

It is required yeah.

Is this all python under the covers? In which case if its using the python taxii libs its supported - so might be an easy fix.

If its something else im happy to take a look.

Scotty

Re: STIX and TAXII support

lmori — Wed, 11 May 2016 05:56:37 GMT

Hi Scotty,

under the cover is mostly python, and yes the lib already supports it. It is just that the option is not exposed via config and it is easy an easy fix.

I have created an ER to track this, minemeld-core ER#13

Re: STIX and TAXII support

ScottyAU — Wed, 11 May 2016 06:02:00 GMT

Wow that was quick!

Should i still have a look or just wait for the ER? (no idea what your backlog/process is like).

Cheers,

Scotty

Re: STIX and TAXII support

lmori — Wed, 11 May 2016 06:14:29 GMT

Hi Scotty,

exposing the option via prototype is easy, it will be a bit more complex exposing the option via the WebUI.

If you are ok in logging in to the VM to upload the cert, this will land in the next minor release. Otherwise, if you need the WebUI immiediately, you will have to wait a bit more.

But if you want to look at the code yourself, you are welcome 🙂 Just check the code in /opt/minemeld/engine/current/lib/python2.7/site-packages/minemeld/ft/taxii.py

When MineMeld will be made Open Source this will be way easier 🙂

Re: STIX and TAXII support

ScottyAU — Mon, 16 May 2016 04:52:43 GMT

Thanks so much mate - and sorry for the delay in response.

I'll configure a new miner off the halitaxii prototype and let you know how i go.

If i create a new prototype - is that local - or does it get pushed back up somwhere for others to see?

Re: STIX and TAXII support

lmori — Mon, 16 May 2016 10:34:23 GMT

Hi Scotty,

1) nothing is shared automatically by MineMeld

2) if you create a new prototype using the NEW button, it will be saved locally in a separate directory. This way it won't be overridden at the next update.

Please, let me know if you need more details about the TAXII prototype.

Luigi

Re: STIX and TAXII support

MGBerkowitz@wellington.com — Tue, 17 May 2016 13:18:54 GMT

This would be very helpful for those of us in the Financial Industry who want to pull in the FS-ISAC feed.

Re: STIX and TAXII support

lmori — Tue, 17 May 2016 13:23:20 GMT

Hi MGBerkowitz,

I will work on it. We already have customers collecting FS-ISAC indicators in Soltra Edge and then using MineMeld to enforce active indicators from Soltra Edge on our NGFW platforms. This way they can conduct manual analysis of indicators on Soltra Edge and use MineMeld to select the active indicators.

I will keep you posted on the progress of this feature.

Thanks,

luigi

Re: STIX and TAXII support

ScottyAU — Mon, 23 May 2016 04:27:47 GMT

Ok so I hacked in certificate support this afternoon. This is just that - a hack (it's not configureable and uses the same cert for any taxii collection).

Once the changes were made i cloned the existing halitaxii prototype and created one with the relevant URL, user creds and collection name. Worked first go!

Does anyone have a list of what STIX vocab is supported by minemeld? It pulled a chunk of data in - but only showed a very small subset of indicators as a result. Guessing it doesn't understand everything we publish. Would be great to get a list so we know what is and isnt supported (and possibly suggest some additions) 🙂

/opt/minemeld/engine/current/lib/python2.7/site-packages/minemeld/ft/taxii.py

def configure(self):
        super(TaxiiClient, self).configure()

        self.discovery_service = self.config.get('discovery_service', None)
        self.username = self.config.get('username', None)
        self.password = self.config.get('password', None)
++        self.key_file = '/opt/certs/browsc-key.pem'
++        self.cert_file = '/opt/certs/browsc-cert.pem'
        self.collection = self.config.get('collection', None)
        self.prefix = self.config.get('prefix', self.name)
        self.ca_file = self.config.get('ca_file', None)
        self.confidence_map = self.config.get('confidence_map', {
            'low': 40,
            'medium': 60,
            'high': 80
        })

    def _build_taxii_client(self):
        result = libtaxii.clients.HttpClient()

        up = urlparse.urlparse(self.discovery_service)

        if up.scheme == 'https':
            result.set_use_https(True)

        if self.username and self.password:
++            result.set_auth_type(libtaxii.clients.HttpClient.AUTH_CERT_BASIC)
--            result.set_auth_type(libtaxii.clients.HttpClient.AUTH_BASIC)
            result.set_auth_credentials({
                'username': self.username,
++                'password': self.password,
--                'password': self.password
++                'key_file': self.key_file,
++                'cert_file': self.cert_file
            })

        if self.ca_file is not None:
            result.set_verify_server(
                verify_server=True,
                ca_file=self.ca_file
            )

        return result

Re: STIX and TAXII support

lmori — Mon, 23 May 2016 06:36:22 GMT

Hi ScottyAU,

that's great ! thanks for testing this. I will include something similar in the next release.

Currently the TAXII Miner supports indicators with observables of type DomainNameObjectType, AddressObjectType, URIObjectType. It can easily be extened to support additional types.

What type of indicators are you receiving via TAXII ?

Thanks,

luigi

Re: STIX and TAXII support

ScottyAU — Tue, 24 May 2016 03:35:50 GMT

Hi Luigi,

This is STIX packages that we (CERT Australia) produce currently and push out to partners. We're looking at minemeld in the event we have any partner companies that want to use it to talk to us.

The STIX elements we use are:

* Package

* Indicator

* CourseOfAction

* TTP

* KillChain / KillChainPhase

Our Indicators typically contain one or more Cybox Observables each of which describes a Cybox Object. Our STIX packages will potentially include the following Cybox Object types:

* Address

* DomainName

* EmailAddress

* EmailMessage

* File

* HTTPSession

* SocketAddress

* URI

* WinRegistryKey

Cheers,

Scotty

Re: STIX and TAXII support

lmori — Tue, 24 May 2016 04:16:25 GMT

Hi Scotty,

I could easily add support for the those additional indicator types, if you could send me an email we can talk about the detailed requirements. My email is lmori@paloaltonetworks.com

Thanks,

luigi

Re: STIX and TAXII support

ScottyAU — Tue, 24 May 2016 04:35:21 GMT

Will do!

Re: STIX and TAXII support

lmori — Wed, 25 May 2016 15:02:28 GMT

For the posterity: client certificates are supported in TAXII miner since MM version 0.9.12

Re: STIX and TAXII support

ScottyAU — Fri, 03 Jun 2016 02:50:31 GMT

Hey Luigi,

Is there anyway for the inital poll to be for a longer historic period?

It just does an hour prior to current time.

So the last year or two of data is not pulled in - becuse the begin and end timestamp is only the previous hour to when the job was run.

Cheers,

SCotty

Re: STIX and TAXII support

lmori — Fri, 03 Jun 2016 07:23:15 GMT

Not yet, but it is a while I wanted to expose it to the config.

ER minemeld-core #18 has been created to track this, it should make into the next minor release.