AWS output/aggregator with unexpected missing indicators

L2 Linker

AWS output/aggregator with unexpected missing indicators

Hello,

 

I have a dev and prod instance of MM.  Noticed both dev and prod behaving the same way, where many indicators were not showing up in output feed for AWS miners.  Config on both is the following:

 

nodes:
  aws_route53_miner:
    inputs: []
    output: true
    prototype: aws.ROUTE53
  aws_ipv4_output:
    inputs:
      - aws_ipv4_aggregator
    output: false
    prototype: stdlib.feedHCWithValue
  aws_s3_miner:
    inputs: []
    output: true
    prototype: minemeldlocal.aws_S3
  aws_ec2_miner:
    inputs: []
    output: true
    prototype: aws.EC2
  aws_amazon_miner:
    inputs: []
    output: true
    prototype: aws.AMAZON
  aws_ipv4_aggregator:
    inputs:
      - aws_cloudfront_miner
      - aws_ec2_miner
      - aws_route53_miner
      - aws_route53_healthchecks_miner
      - aws_amazon_miner
      - aws_s3_miner
    output: true
    prototype: stdlib.aggregatorIPv4Generic
  aws_cloudfront_miner:
    inputs: []
    output: true
    prototype: aws.CLOUDFRONT
  aws_route53_healthchecks_miner:
    inputs: []
    output: true
    prototype: aws.ROUTE53_HEALTHCHECKS

Dev/Prod both showed the same # of indicators on Nodes tab:

 

aws_amazon_miner 511

aws_cloudfront_miner 35

aws_ec2_miner 222

aws_route53_healthchecks_miner 16

aws_route53_miner 2

aws_s3_miner 62

 

aws_ipv4_aggregator 323

aws_ipv4_output 346

 

I made a change in dev to take the aws_amazon_miner with 511 indicators straight to an output.

 

nodes:
  aws_test_full_output:
    inputs:
      - aws_amazon_miner
    output: false
    prototype: stdlib.feedHCWithValue

The result is that the issue is not seen on this output, but the original aggregator and output nodes also had the issue go away for the time being:

 

aws_ipv4_aggregator 848

aws_ipv4_output 577

aws_test_full_output 511

 

Was planning to go production with this AWS output next week.  Need to vet out this issue asap.

 

Attaching engine logs from both instances as well as copys of output feeds in multiple versions.

 

prod  0.9.36

dev 0.9.38

 

L7 Applicator

Re: AWS output/aggregator with unexpected missing indicators

Hi @andrew.stanton,

thanks, this is a bug and will be fixed in the next release. Details here: https://github.com/PaloAltoNetworks/minemeld-core/issues/213

 

A workaround is forcing a flush and an update on the existing miner you just added to the aggregator:

$ /opt/minemeld/engine/current/bin/mm-console signal flush aws_amazon_miner
$ /opt/minemeld/engine/current/bin/mm-console hup aws_amazon_miner
Like what you see?

Show your appreciation!

Click Like if a post is helpful to you or if you just want to show your support.

Click Accept as Solution to acknowledge that the answer to your question has been provided.

The button appears next to the replies on topics you’ve started. The member who gave the solution and all future visitors to this topic will appreciate it!

These simple actions take just seconds of your time, but go a long way in showing appreciation for community members and the Live Community as a whole!

The Live Community thanks you for your participation!