How does one create an output filter to exclude IPv4 indicators in a CIDR range?

[EdwinD]

I have various miners.   Various miners are connected to various aggregators which are inturn connected in various ways to different types of output.

Some of these miners receive RFC1918 IPv4 indicators.   These are aggregated and send to outputs.   I'm attempted to have one output which will contain these RFC1918 addresses while another does not.   

I've created a new prototype based off of minemeld.ft.redis.RedisSet accepting only indicator types IPv4.   The config includes this:

store_value: true
whitelist_prefixes:
- wl

I cloned it and pointed it to clone of processor stdlib.aggregatorIPv4Outbound.   The input for this processor includes my wlWhiteListIPv4 as well as my miner.  This wlWhiteListIPv4 node has manual indicators which include RFC1918 CIDR blocks along with one IP address (for testing.)   This, it includes 10.0.0.0/8 as well as a single host within that range.  It appears to me that while the single host works in this case, the CIDR does not.   With two nearly identical outputs configured where one is  using the whitelist and another not, I see this single host only in the later output.   However, other hosts in the 10.0.0.0/8 CIDR are in both.    

How do I go about adding a miner with CIDRs as a whitelist to an output node?    While in my example above I'm talking about RFC1918 addresses, there are additional cases where I want to exclude from certain output nodes the ranges collected from a miner.  

[xhoms]

@EdwinD : It is working in my case. Are you using a miner based on the 'localdb.Miner' to host the RFC1928 CIDR ranges?

[EdwinD]

No, although I do have a WildFireEvent miner cloned from localDB as per the article "Using-MineMeld-as-a-Incident-Response-Platform".   Is localDB what I should be using?   

[EdwinD]

I believe I see what I had wrong.    Because I hwave whitelist_prefixes set to wl, the processor name itself must also begin with wl.  I had only named the miner begining with wl.

So, I configured a processor with a name begining with wl and bound it to two inputs, the wl miner containing my RFC1918 addresses along with my syslog+7 days miner.  This worked.   Within this same config I have an identical processer with the exception of the name, it does not start with wl.  The output node is identical except it points to this processor.  This one has the RFC1918 addresses.   The point being that with only the name of the processor changing I can see the whitelist working.  

I think the following is the relevant config.   I have 300+  lines in the config ATM, so I'm not posting it all.

The end result in this case is that I have PanOS syslog output feeding into MineMeld while whitelisting RFC1918 addresses.  Longer term I could use external dynamic IPv4 lists, such as a list of CloudFlare addresses,  as a feed into a whitelist that I aggregate against the PanOS syslog entries to generate output feeds that I can use in specific dynamic PanOS firewall rules.  After enough hits from an IP, if it isn't on a known list, completely block that IP at every firewall at all locations for a period of time.  This is just one example of many I'm considering.

  syslogMiner7Days:
    inputs: []
    output: true
    prototype: minemeldlocal.syslogMiner7Days
  aggregatorIPv4GenericStaticWhiteList:
    inputs:
      - wlWhiteListIPv4
    output: true
    prototype: stdlib.aggregatorIPv4Generic
  wlWhiteListIPv4:
    inputs: []
    output: true
    prototype: stdlib.listIPv4Generic
  localSyslog7Days:
    inputs:
      - syslogMiner7Days
      - wlWhiteListIPv4
    output: true
    prototype: stdlib.localSyslog
  wlListIPv4Generic-testingWhiteList20171222b:
    inputs:
      - wlWhiteListIPv4
      - syslogMiner7Days
    output: true
    prototype: stdlib.aggregatorIPv4Generic
  feedIPv4TestingWhiteList:
    inputs:
      - wlListIPv4Generic-testingWhiteList20171222b
    output: false
    prototype: minemeldlocal.feedIPv4TestingWhiteList

[lmori]

Hi @EdwinD,

are you trying to create a feed out of MineMeld to be used as whitelist? or would you like to whitelist some IPs from the indicators extraced from syslog?

Thanks,

luigi

[EdwinD]

>> whitelist some IPs from the indicators extraced from syslog?

While not excluding them at the time of extract.  Not all feeds should have these whitelist entries excluded from their final output.