Consolidating or aggregating IP addresses in Processor
cancel
Showing results for 
Search instead for 
Did you mean: 

Consolidating or aggregating IP addresses in Processor

L0 Member

Apologies if this question has been asked before, I searched the board but couldn't see anything that stood out

 

I'm consuming the SANS/IRC list of IP addresses attributed to Internet Security Researchers, in an attempt to cut-down on false-positive threat alerts in customer networks. The Miner I wrote works fine, and pulls down about ~6700 IP addresses.

 

When passed into a Processor (Cloned from 'stdlib.aggregatorIPv4Generic', and of the same class 'minemeld.ft.ipop.AggregateIPv4FT') all addresses are getting sent to an output.

 

The problem is that because the list is gathered automatically, it has one IP per-line. Example;

102.165.30.0-102.165.30.0
102.165.30.1-102.165.30.1

102.165.30.2-102.165.30.2

102.165.30.3-102.165.30.3

...

102.165.30.255-102.165.30.255

 

I feel that added ~6700 entries into an EDL will be unneccessarily taxing on the firewall.

Since I'm putting in a top-level firewall rule to 'drop' packets coming from these IPs, the firewall will have to match the incoming packets IP to all ~6700 possibilities - whereas if I could consolidate the IPs (for example the above consolidates into 102.165.30.0/24) then the number of matches greatly decreases

 

I realise that 'aggregator' in the Processors name refers to the ability to 'aggregate' from multiple miners into one processor. But is there an ability to aggregate (/consolidate) IP addresses inside a Processor?

 

If this can't be done in Minemeld, then I may have to write a Python parser to pull down the list and consolidate manually - but at that point Minemeld becomes irrelevant, as if I'm hosting the output of the Python script somewhere I can just point the firewall to that instead

 

Thanks

2 REPLIES 2

L0 Member

Have you checked out this form Lmori, in https://live.paloaltonetworks.com/t5/minemeld-discussions/miner-to-collect-aws-ip/td-p/75925

Yes, add use the following format for the URL feed:

https://<minemeld>/feeds/<aws feed>?tr=1

 

See here for additional details:

https://live.paloaltonetworks.com/t5/MineMeld-Articles/Parameters-for-the-output-feeds/ta-p/146170

Hi Dereje

 

Thanks for the link, that's useful info

Unfortunately ?tr=1 won't consolidate subnets, it justr translates each line

 

So

102.165.30.0-102.165.30.0
102.165.30.1-102.165.30.1
102.165.30.10-102.165.30.10

Becomes

102.165.30.0
102.165.30.1
102.165.30.10

 

Appreciate the reply though

Like what you see?

Show your appreciation!

Click Like if a post is helpful to you or if you just want to show your support.

Click Accept as Solution to acknowledge that the answer to your question has been provided.

The button appears next to the replies on topics you’ve started. The member who gave the solution and all future visitors to this topic will appreciate it!

These simple actions take just seconds of your time, but go a long way in showing appreciation for community members and the LIVEcommunity as a whole!

The LIVEcommunity thanks you for your participation!