Polling JSON Format for Okta

[doliver1]

I am trying to create a prototype for a Miner that pulls IP's from a JSON formatted file. I have looked at the documentation for setting up a JSON miner (https://live.paloaltonetworks.com/t5/MineMeld-Articles/Using-MineMeld-to-extract-indicators-from-a-g...) and I am having little luck as the error messages in the logs are saying there is a parsing error.

File "/opt/minemeld/engine/0.9.48.post1/local/lib/python2.7/site-packages/minemeld/ft/basepoller.py", line 606, in _polling_loop
ipairs = process_item(item)
File "/opt/minemeld/engine/0.9.48.post1/local/lib/python2.7/site-packages/minemeld/ft/json.py", line 192, in _process_item
indicator = item[self.indicator]
TypeError: string indices must be integers

My Config looks like the following:

age_out:
default: null
interval: 257
sudden_death: true
attributes:
confidence: 100
share_level: green
type: IPv4
extractor: preview_cell_2
indicator: ip_ranges
source_name: aws.S3
url: https://s3.amazonaws.com/okta-ip-ranges/ip_ranges.json

Can anyone assist?

[xhoms · Accepted solution]

Hi @doliver1,

your extractor/indicator combination produces an array of strings instead of an array of objects (you can verify it at http://jmespath.org/)

[
  "3.120.18.145/32",
  "3.120.73.242/32",
  "3.120.77.98/32",
  ...
]

You need the extractor to generate an array of objects and tell with the indicator which property in each object inside the extracted array contains the indicator.

Use the extrator "preview_cell_2.ip_ranges[].{indicator:@}" that will produce the following array of objects:

[
  {
    "indicator": "3.120.18.145/32"
  },
  {
    "indicator": "3.120.73.242/32"
  },
  {
    "indicator": "3.120.77.98/32"
  },
...
]

and use as "indicator" as indicator or just remove it as that is its default value.

[doliver1]

@xhoms thanks for the catch on that. It works perfectly now.

[KengSeng]

Hi Xavier,

I tried to use this as extrator "*.ip_ranges[].{indicator:@}", but the AF minemeld doesnt allow me to click OK. I would like to extract everything from the Okta website.

Regards,

Keng Seng

[KengSeng]

I used the ubuntu VM is working.

age_out	default: null interval: 257 sudden_death: true
attributes	confidence: 100 share_level: green type: IPv4
extractor	*.ip_ranges[].{indicator:@}
indicator	indicator
url	https://s3.amazonaws.com/okta-ip-ranges/ip_ranges.json

[xhoms]

Hi @KengSeng,

AF-MM has a different UI and may have a different YAML validator. Try quoting the string ("*.ip_ranges[].{indicator:@}"). It should work with that.

[KengSeng]

Thanks Xavier, it works!!!

[niuk]

 ("*.ip_ranges[].{indicator:@}") doesnt pull any indicators for me , tried also  (*.ip_ranges[].{indicator:@}), same thing I am using VERSION: 0.9.50  and 0.9.52