Consolidating or aggregating IP addresses in Processor

[sam_miller]

Apologies if this question has been asked before, I searched the board but couldn't see anything that stood out

I'm consuming the SANS/IRC list of IP addresses attributed to Internet Security Researchers, in an attempt to cut-down on false-positive threat alerts in customer networks. The Miner I wrote works fine, and pulls down about ~6700 IP addresses.

When passed into a Processor (Cloned from 'stdlib.aggregatorIPv4Generic', and of the same class 'minemeld.ft.ipop.AggregateIPv4FT') all addresses are getting sent to an output.

The problem is that because the list is gathered automatically, it has one IP per-line. Example;

102.165.30.0-102.165.30.0
102.165.30.1-102.165.30.1

102.165.30.2-102.165.30.2

102.165.30.3-102.165.30.3

...

102.165.30.255-102.165.30.255

I feel that added ~6700 entries into an EDL will be unneccessarily taxing on the firewall.

Since I'm putting in a top-level firewall rule to 'drop' packets coming from these IPs, the firewall will have to match the incoming packets IP to all ~6700 possibilities - whereas if I could consolidate the IPs (for example the above consolidates into 102.165.30.0/24) then the number of matches greatly decreases

I realise that 'aggregator' in the Processors name refers to the ability to 'aggregate' from multiple miners into one processor. But is there an ability to aggregate (/consolidate) IP addresses inside a Processor?

If this can't be done in Minemeld, then I may have to write a Python parser to pull down the list and consolidate manually - but at that point Minemeld becomes irrelevant, as if I'm hosting the output of the Python script somewhere I can just point the firewall to that instead

Thanks

[Dereje]

Have you checked out this form Lmori, in https://live.paloaltonetworks.com/t5/minemeld-discussions/miner-to-collect-aws-ip/td-p/75925

Yes, add use the following format for the URL feed:

https://<minemeld>/feeds/<aws feed>?tr=1

See here for additional details:

https://live.paloaltonetworks.com/t5/MineMeld-Articles/Parameters-for-the-output-feeds/ta-p/146170

[Dereje]

Yes, add use the following format for the URL feed:

https://<minemeld>/feeds/<aws feed>?tr=1

See here for additional details:

https://live.paloaltonetworks.com/t5/MineMeld-Articles/Parameters-for-the-output-feeds/ta-p/146170

[sam_miller]

Hi Dereje

Thanks for the link, that's useful info

Unfortunately ?tr=1 won't consolidate subnets, it justr translates each line

So

102.165.30.0-102.165.30.0
102.165.30.1-102.165.30.1
102.165.30.10-102.165.30.10

Becomes

102.165.30.0
102.165.30.1
102.165.30.10

Appreciate the reply though