topic Re: AlienVault taxii miner versus prebuilt reputation data miner in General Topics

AlienVault taxii miner versus prebuilt reputation data miner

chirss — Wed, 17 May 2017 15:21:58 GMT

AlientVault has the OTX with a taxii feed configuration which looks like it could be handy. However the miner for the alienvault reputation has a link which 404's. Does anyone have any idea if this overlaps?

Also the AlienVault taxii feed would require an API key, I'm not entirely sure how to set up a new miner for taxii which requires an API key.

Re: AlienVault taxii miner versus prebuilt reputation data miner

lmori — Thu, 18 May 2017 06:29:32 GMT

Hi @chirss,

the link for the alienvault reputation works for me.

Do you have more details about the OTX TAXII feed ? a config guide ?

Thanks,

luigi

Re: AlienVault taxii miner versus prebuilt reputation data miner

chrisdoman — Thu, 18 May 2017 11:23:55 GMT

Hey guys,

Taxii url for otx: https://otx.alienvault.com/taxii/discovery

If you don't enter any API key it will still work - you'll just get the default feeds.

If you'd like to customise the feeds, then you should enter you api key in the username field.

Docs: https://www.alienvault.com/blogs/security-essentials/otx-is-now-a-free-stix-taxii-server

Cheers,

Re: AlienVault taxii miner versus prebuilt reputation data miner

lmori — Fri, 19 May 2017 12:15:17 GMT

I have done a quick test on the public feed:

- select an existing TAXII prototype (anything starting with hailataxii)

- click NEW

- copy & paste the following config in the CONFIG section

age_out:
    default: 30d
    sudden_death: false
attributes:
    confidence: 30
    share_level: green
client_credentials_required: false
collection: user_AlienVault
discovery_service: https://otx.alienvault.com/taxii/discovery
source_name: alienvault.user_AlienVault
initial_interval: 7d

- create a new node with CLONE from the new prototype

Result

There is a bug in the AV TAXII Server, it announces indicators in STIX 1.1 format while the content is using STIX 1.1.1. MineMeld throws an error. I have done a quick test and after removing the check the indicators are successfully extracted.

Do you know someone in AlienVault we can work with to fix this ? Otherwise I can just add a flag in the TAXII Miner config to disable the check.

Re: AlienVault taxii miner versus prebuilt reputation data miner

chirss — Fri, 19 May 2017 14:25:59 GMT

One of the people involved is actually fairly active in responding on the twitters. I'll reply to him and point him to this thread.

Re: AlienVault taxii miner versus prebuilt reputation data miner

chrisdoman — Fri, 19 May 2017 14:56:30 GMT

Thanks @chrisf

Looks like we've fixed this already in our development environment, and a fix should be released next week.

It'd be great to test it first though - if anyone here can e-mail us at otx-support@alienvault.com we can send you the development Taxii feed address to check that everything is working correctly.

Cheers 🙂

Re: AlienVault taxii miner versus prebuilt reputation data miner

lmori — Fri, 19 May 2017 15:01:48 GMT

Hi @chrisdoman,

could you send me the address at lmori@paloaltonetworks.com ? I would be happy to test and adapt MineMeld if needed.

Thanks!

luigi

Re: AlienVault taxii miner versus prebuilt reputation data miner

chirss — Fri, 19 May 2017 18:19:45 GMT

Is there benefit to going through the steps for the api key?

Re: AlienVault taxii miner versus prebuilt reputation data miner

chrisdoman — Tue, 23 May 2017 22:20:22 GMT

@chirss Yes if you sign up then you can choose to follow additional users, beyond the default main account. That means you get more indicators.

I haven't checked (I'm away at the moment) but I believe this should work now

Re: AlienVault taxii miner versus prebuilt reputation data miner

lmori — Fri, 26 May 2017 09:17:00 GMT

Just checked and it works now. For the public feed follow the procedure:

- in the prototype library, click on one of the TAXII prototypes (like hailataxii.guest_Abuse_ch) and press NEW

- copy & paste the following config in the CONFIG part of the prototype. Press OK

age_out:
    default: 30d
    sudden_death: false
attributes:
    confidence: 30
    share_level: green
client_credentials_required: true
collection: user_AlienVault
discovery_service: https://otx.alienvault.com/taxii/discovery
initial_interval: 7d
source_name: alienvault.otx

- CLONE the new prototype into a node and COMMIT

After the COMMIT you can click on the new Miner under NODES to specify the credentials. For the public feed you can use whatever you want.

You can use the same prototype with different collection names to retrieve indicators from the private feeds, you'll need to specify the API Key as username in the Miner

Re: AlienVault taxii miner versus prebuilt reputation data miner

chirss — Mon, 19 Jun 2017 17:38:30 GMT

Thanks this seems to work. I have it pulling some data. Not any feeds I subscribe to though.

Re: AlienVault taxii miner versus prebuilt reputation data miner

chirss — Fri, 08 Sep 2017 15:27:51 GMT

How could I go about changing this to mine sha256 indicators out of otx?

Re: AlienVault taxii miner versus prebuilt reputation data miner

lmori — Sun, 17 Sep 2017 09:14:17 GMT

Hi @chirss,

just tested this and it works for me:

Re: AlienVault taxii miner versus prebuilt reputation data miner

chirss — Mon, 02 Oct 2017 17:54:10 GMT

Thanks. I'm able to clone a miner and set it to sha256 (or sha1) and it pulls 315 indicators. What I'm trying to figure out is how it relates to subscriptions in otx, if anyone knows.

Thanks for confirming, I was able to at least pull data once I made a miner for them (noticed I wasn't looking for hashes in the miner itself)

Re: AlienVault taxii miner versus prebuilt reputation data miner

lmori — Wed, 04 Oct 2017 22:49:18 GMT

Good question, I hope @chrisdoman could help here

Re: AlienVault taxii miner versus prebuilt reputation data miner

chirss — Wed, 24 Oct 2018 17:19:06 GMT

I'm back on this, a year later. Trying to relate subscribing to a person or pulse to how it gets pulled by the miner, if it does.

Re: AlienVault taxii miner versus prebuilt reputation data miner

Carlos_Gomes — Thu, 20 Jun 2019 20:32:10 GMT

Just followed the directions here and on mine the miner under nodes where i go to set the API key then asks for password? But there is no password and it will not work. What would i enter there?

Re: AlienVault taxii miner versus prebuilt reputation data miner

Carlos_Gomes — Thu, 11 Jul 2019 19:02:01 GMT

Got this workind. any txt will do it will be ignored anyways. Go figure.

Has anyone successfully stripped http and https from a taxii Feed so that PaloAlto FWs can block those URLs? Currently aggregating the URLs into an output works and PAs can pull them into EDLs however it is pulling them with http and https which PAs then are not able to block those objects, according to PA article any url object cannot contain http:// or https://

I found the example here https://live.paloaltonetworks.com/t5/MineMeld-Articles/Using-MineMeld-to-Create-a-Custom-Miner/tac-p...

It shows how to do it against a csv/txt list but i have not been able to find it for the taxi client feed. I applied the same terminology but that did not work. Anyone else has been able to do this and can share some insight?

Re: AlienVault taxii miner versus prebuilt reputation data miner

joris_vd — Wed, 21 Aug 2019 15:29:10 GMT

Hi,

In the EDL Create List window on the firewall you can add the "Source"...

It should be like this : https://minemeldserver.internal/feeds/OTXURLOutput?v=panosurl

Cheers

Joris

Re: AlienVault taxii miner versus prebuilt reputation data miner

Carlos_Gomes — Wed, 04 Sep 2019 21:50:18 GMT

@joris_vd thank you for responding however i have that part done it is not what i am asking. What i am asking is how you can have minemeld strip the http and https from the list of indicators as the PaloAlto firewalls are ingesting the list containing http and https for all urls and it is not supported, it needs to be just like google.com for example rather than what minemeld list is holding http://google.com

With a txt/csv file minemeld feed you can manipulate it and tell it to drop the http and https from all URL indicators however i have not been able to find it for taxifeed.