topic Re: Miner to collect AWS IP in General Topics

Miner to collect AWS IP

coldstone1 — Wed, 06 Apr 2016 21:29:29 GMT

I would like to setup a new Miner to collect AWS Ips from the following http://docs.aws.amazon.com/general/latest/gr/aws-ip-ranges.html, using URL https://ip-ranges.amazonaws.com/ip-ranges.json. How would that be accomlished, I can't seem to locate a good example.

Re: Miner to collect AWS IP

lmori — Wed, 06 Apr 2016 23:08:27 GMT

Hi coldstone1,

there is no Miner for JSON, but it could be easily added. I have created Enachement Request #7 to track this request

Re: Miner to collect AWS IP

lmori — Tue, 19 Apr 2016 17:35:18 GMT

Implemented in MineMeld 0.9.9:

https://live.paloaltonetworks.com/t5/MineMeld-Discussions/What-s-new-in-MineMeld-0-9-9/m-p/76690#U76690

Re: Miner to collect AWS IP

mr.linus — Mon, 21 Nov 2016 14:24:55 GMT

I can not get the AWS Ips in my output.

My miner gets the IPs, the aggregator gets the IPs, but the outout stays empty.

AWS-Miner:
CLASS minemeld.ft.json.SimpleJSON
PROTOTYPE aws.AMAZON
STATE STARTED
LAST RUN 2016-11-21 15:19:42 +0100 SUCCESS
# INDICATORS 457

AWS: aggegator:
CLASS minemeld.ft.ipop.AggregateIPv4FT
PROTOTYPE stdlib.aggregatorIPv4Inbound
STATE STARTED
# INDICATORS 457

AWS: Output
CLASS minemeld.ft.redis.RedisSet
PROTOTYPE stdlib.feedHCGreen
STATE STARTED
FEED BASE URL https://xxx/feeds/AWS-IPv4-output
TAGS
# INDICATORS 0

Am I using the wrong nodes?

Re: Miner to collect AWS IP

lmori — Tue, 22 Nov 2016 17:27:30 GMT

Hi @mr.linus,

the problem is the output node, as AWS IPs have not share_level green (and that's an oversight) they are not accepted by feedHCGreen. Use an output node based on stdlib.feedHCWithValue please.

Re: Miner to collect AWS IP

mr.linus — Thu, 24 Nov 2016 09:02:45 GMT

Hi,

Just found it. I adjusted the miner and now I get the output.

Thanks for the tip

Re: Miner to collect AWS IP

InfoSecFairway — Wed, 09 Aug 2017 17:32:25 GMT

Thanks for the helpful post, is there a way for the output to be IP ranges in CIDR format, instead of it listing the first IP and the last IP within the range?

Thank you.

Re: Miner to collect AWS IP

lmori — Mon, 14 Aug 2017 09:47:35 GMT

Yes, add use the following format for the URL feed:

https://<minemeld>/feeds/<aws feed>?tr=1

See here for additional details:

https://live.paloaltonetworks.com/t5/MineMeld-Articles/Parameters-for-the-output-feeds/ta-p/146170