Filter items from source feed

Announcements

Changes to the LIVEcommunity experience are coming soon... Here's what you need to know.

Reply
deanm
L2 Linker

Filter items from source feed

One of the feeds I would like to import is the alienvault feed.  However, I only want a subset of the IPs listed.  I have tried using a regex with a transform to limit the results, but the miner is still showing an indicator count of 54,000.

 

I cloned the alienvault prototype and changed it to this:

 

    my_alienvaultreputation:

        class: minemeld.ft.csv.CSVFT

        config:

            attributes:

                confidence: 80

                share_level: green

                type: IPv4

            delimiter: '#'

            fieldnames:

            - indicator

            - alienvault_reliability

            - alienvault_risk

            - alienvault_type

            indicator:

                regex: '([0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3})(.*Malicious Host\#(US|CN))'

                transform: '\1'

            interval: 3600

            source_name: alienvault.reputation

            url: http://reputation.alienvault.com/reputation.data

        description: Malicious US and Canada only alienvault reputation entries

        development_status: EXPERIMENTAL

        indicator_types:

        - IPv4

        node_type: miner

        tags:

        - OSINT

        - ShareLevelGreen

   

The regex itself works, at least in Sublime text when I do a regex search of the alienvault reputation list, which shows approximately 8,000 matches.

 

Is this not possible or is something wrong with the prototype?

 

Thanks,

 

Mike

1 ACCEPTED SOLUTION
lmori
L7 Applicator

Hi @deanm,

I guess the reason is that filters are applied in order and the first matching is used (it works as a traditional firewall rulebase). In your case your rulebase accepts: indicators in US, indicators in CA and indicators of type "Malicious Host" (even if they are not in US or CA). If you want to use the type as additional selector you should use this:

alienvault_reputation-Malicious_US-CA:
        class: minemeld.ft.csv.CSVFT
        config:
            attributes:
                confidence: 80
                share_level: green
                type: IPv4
            delimiter: '#'
            fieldnames:
            - indicator
            - alienvault_reliability
            - alienvault_risk
            - alienvault_type
            - alienvault_country
            outfilters:
            -   actions:
                - accept
                conditions:
                - alienvault_country == 'US'
- alienvault_type == 'Malicious Host'
name: accept US - actions: - accept conditions: - alienvault_country == 'CA'
- alienvault_type == 'Malicious Host' name: accept CA - actions: - drop name: drop all interval: 3600 source_name: alienvault.reputation url: http://reputation.alienvault.com/reputation.data description: Malicious US and CA alienvault hosts development_status: EXPERIMENTAL indicator_types: - IPv4 node_type: miner tags: - OSINT - ShareLevelGreen

 

Please note that you will still see 55K indicators in the Miner, but only a subset of them should be emitted to the attached processors - you can check the UPDATE.RX counter on the processor to double check this.

View solution in original post

Like what you see?

Show your appreciation!

Click Like if a post is helpful to you or if you just want to show your support.

Click Accept as Solution to acknowledge that the answer to your question has been provided.

The button appears next to the replies on topics you’ve started. The member who gave the solution and all future visitors to this topic will appreciate it!

These simple actions take just seconds of your time, but go a long way in showing appreciation for community members and the LIVEcommunity as a whole!

The LIVEcommunity thanks you for your participation!