Polling JSON Format for Okta

Reply
L0 Member

Polling JSON Format for Okta

I am trying to create a prototype for a Miner that pulls IP's from a JSON formatted file. I have looked at the documentation for setting up a JSON miner (https://live.paloaltonetworks.com/t5/MineMeld-Articles/Using-MineMeld-to-extract-indicators-from-a-g...) and I am having little luck as the error messages in the logs are saying there is a parsing error.

 

File "/opt/minemeld/engine/0.9.48.post1/local/lib/python2.7/site-packages/minemeld/ft/basepoller.py", line 606, in _polling_loop
ipairs = process_item(item)
File "/opt/minemeld/engine/0.9.48.post1/local/lib/python2.7/site-packages/minemeld/ft/json.py", line 192, in _process_item
indicator = item[self.indicator]
TypeError: string indices must be integers

 

My Config looks like the following:

age_out:
default: null
interval: 257
sudden_death: true
attributes:
confidence: 100
share_level: green
type: IPv4
extractor: preview_cell_2
indicator: ip_ranges
source_name: aws.S3
url: https://s3.amazonaws.com/okta-ip-ranges/ip_ranges.json

 

Can anyone assist?

Tags (1)
L5 Sessionator

Re: Polling JSON Format for Okta

Hi @doliver1,

 

your extractor/indicator combination produces an array of strings instead of an array of objects (you can verify it at http://jmespath.org/)

 

[
  "3.120.18.145/32",
  "3.120.73.242/32",
  "3.120.77.98/32",
  ...
]

You need the extractor to generate an array of objects and tell with the indicator which property in each object inside the extracted array contains the indicator.

 

Use the extrator "preview_cell_2.ip_ranges[].{indicator:@}" that will produce the following array of objects:

 

[
  {
    "indicator": "3.120.18.145/32"
  },
  {
    "indicator": "3.120.73.242/32"
  },
  {
    "indicator": "3.120.77.98/32"
  },
...
]

and use as "indicator" as indicator or just remove it as that is its default value.

L0 Member

Re: Polling JSON Format for Okta

@xhoms thanks for the catch on that. It works perfectly now.

L1 Bithead

Re: Polling JSON Format for Okta

Hi Xavier,

 

I tried to use this as extrator "*.ip_ranges[].{indicator:@}", but the AF minemeld doesnt allow me to click OK. I would like to extract everything from the Okta website.

 

Regards,

Keng Seng

L1 Bithead

Re: Polling JSON Format for Okta

I used the ubuntu VM is working.

 

age_out
  • default: null
  • interval: 257
  • sudden_death: true
attributes
  • confidence: 100
  • share_level: green
  • type: IPv4
extractor *.ip_ranges[].{indicator:@}
indicator indicator
url https://s3.amazonaws.com/okta-ip-ranges/ip_ranges.json
L5 Sessionator

Re: Polling JSON Format for Okta

Hi @KengSeng,

 

AF-MM has a different UI and may have a different YAML validator. Try quoting the string ("*.ip_ranges[].{indicator:@}"). It should work with that.

Highlighted
L1 Bithead

Re: Polling JSON Format for Okta

Thanks Xavier, it works!!!

L3 Networker

Re: Polling JSON Format for Okta

 ("*.ip_ranges[].{indicator:@}") doesnt pull any indicators for me , tried also  (*.ip_ranges[].{indicator:@}), same thing I am using VERSION: 0.9.50  and 0.9.52

Like what you see?

Show your appreciation!

Click Like if a post is helpful to you or if you just want to show your support.

Click Accept as Solution to acknowledge that the answer to your question has been provided.

The button appears next to the replies on topics you’ve started. The member who gave the solution and all future visitors to this topic will appreciate it!

These simple actions take just seconds of your time, but go a long way in showing appreciation for community members and the Live Community as a whole!

The Live Community thanks you for your participation!