Miner to collect AWS IP

[coldstone1]

I would like to setup a new Miner to collect AWS Ips from the following http://docs.aws.amazon.com/general/latest/gr/aws-ip-ranges.html, using URL https://ip-ranges.amazonaws.com/ip-ranges.json.    How would that be accomlished, I can't seem to locate a good example.

[lmori · Accepted solution]

Hi coldstone1,

there is no Miner for JSON, but it could be easily added. I have created Enachement Request #7 to track this request

[lmori · Accepted solution]

Hi coldstone1,

there is no Miner for JSON, but it could be easily added. I have created Enachement Request #7 to track this request

[lmori]

Implemented in MineMeld 0.9.9: 

https://live.paloaltonetworks.com/t5/MineMeld-Discussions/What-s-new-in-MineMeld-0-9-9/m-p/76690#U76...

[mr.linus]

I can not get the AWS Ips in my output.

My miner gets the IPs, the aggregator gets the IPs, but the outout stays empty.

AWS-Miner:
CLASS minemeld.ft.json.SimpleJSON
PROTOTYPE aws.AMAZON
STATE STARTED
LAST RUN 2016-11-21 15:19:42 +0100 SUCCESS
# INDICATORS 457

AWS: aggegator:
CLASS minemeld.ft.ipop.AggregateIPv4FT
PROTOTYPE stdlib.aggregatorIPv4Inbound
STATE STARTED
# INDICATORS 457

AWS: Output
CLASS minemeld.ft.redis.RedisSet
PROTOTYPE stdlib.feedHCGreen
STATE STARTED
FEED BASE URL https://xxx/feeds/AWS-IPv4-output
TAGS
# INDICATORS 0

Am I using the wrong nodes?

[lmori]

Hi @mr.linus,

the problem is the output node, as AWS IPs have not share_level green (and that's an oversight) they are not accepted by feedHCGreen. Use an output node based on stdlib.feedHCWithValue please. 

[mr.linus]

Hi,

Just found it. I adjusted the miner and now I get the output.

Thanks for the tip

[InfoSecFairway]

Thanks for the helpful post, is there a way for the output to be IP ranges in CIDR format, instead of it listing the first IP and the last IP within the range?

Thank you.

[lmori]

Yes, add use the following format for the URL feed:

https://<minemeld>/feeds/<aws feed>?tr=1

See here for additional details:

https://live.paloaltonetworks.com/t5/MineMeld-Articles/Parameters-for-the-output-feeds/ta-p/146170