Custom url feeds

chirss — Wed, 21 Sep 2016 21:44:19 GMT

Is there any sort of documentation surrounding things like adding a custom url in? I'm thinking I'll have to dig into the file system but was wondering if there is anything documented as to what to do.

Say I have an ip list at http://somefancywebsite.com/directory/badiplist.txt I wanted to throw into the mix here. Is there an easy way to do this in the gui or do I need to go hit the file system?

Re: Custom url feeds

lmori — Thu, 22 Sep 2016 14:13:44 GMT

That is usually possible by defining a new prototype using the Web UI.

Do you have a specific example ?

Re: Custom url feeds

chirss — Thu, 22 Sep 2016 17:46:02 GMT

Nothing specific. More trying to understand how the system works.

Re: Custom url feeds

lmori — Thu, 22 Sep 2016 22:01:14 GMT

To add a new feed you should start from the following details:

what are the format and protocol used by the feed ?
what expiration policy should I apply to the indicators ?
what confidence level should I use for the indicators ?

Question 1) defines the class of Miner you want to use. Currently there are classes supporting plain text feeds over HTTP/HTTPS, JSON over HTTP/HTTPS, CSV over HTTP/HTTPS, STIX/TAXII, and a number of other classes for specific public or commercial API. If the protocol and format used by the feed are not covered by one of the existing classes you should write your own Python class. Most of the times it's pretty easy, details here: https://github.com/PaloAltoNetworks/minemeld/wiki/How-To-Write-a-Simple-Miner

If instead the protocol and format are already covered, you don't need to write a single line of code. You can just write a prototype, i.e. a config for the Miner. In the Web UI go under CONFIG and click the browse button (the 3 stackd lines). Select a prototype for a feed similar to the one you want to add and click on the NEW button in the top right corner. This will create a private copy of the prototype you can modify. Now you can change the config of the Miner and specify new parameters, like URL, age out policy, confidence level, new attributes, ...

Additional details about prototypes here:

https://github.com/PaloAltoNetworks/minemeld-core/blob/master/docs/nodeconfig.rst

https://live.paloaltonetworks.com/t5/MineMeld-Articles/What-is-in-a-MineMeld-node/ta-p/72046

topic Custom url feeds in General Topics

Custom url feeds

Re: Custom url feeds

Re: Custom url feeds

Re: Custom url feeds