topic Re: Polling JSON Format for Okta in General Topics

Polling JSON Format for Okta

doliver1 — Wed, 17 Oct 2018 15:30:32 GMT

I am trying to create a prototype for a Miner that pulls IP's from a JSON formatted file. I have looked at the documentation for setting up a JSON miner (https://live.paloaltonetworks.com/t5/MineMeld-Articles/Using-MineMeld-to-extract-indicators-from-a-generic-API/ta-p/218757) and I am having little luck as the error messages in the logs are saying there is a parsing error.

File "/opt/minemeld/engine/0.9.48.post1/local/lib/python2.7/site-packages/minemeld/ft/basepoller.py", line 606, in _polling_loop
ipairs = process_item(item)
File "/opt/minemeld/engine/0.9.48.post1/local/lib/python2.7/site-packages/minemeld/ft/json.py", line 192, in _process_item
indicator = item[self.indicator]
TypeError: string indices must be integers

My Config looks like the following:

age_out:
default: null
interval: 257
sudden_death: true
attributes:
confidence: 100
share_level: green
type: IPv4
extractor: preview_cell_2
indicator: ip_ranges
source_name: aws.S3
url: https://s3.amazonaws.com/okta-ip-ranges/ip_ranges.json

Can anyone assist?

Re: Polling JSON Format for Okta

xhoms — Fri, 26 Oct 2018 08:47:51 GMT

Hi @doliver1,

your extractor/indicator combination produces an array of strings instead of an array of objects (you can verify it at http://jmespath.org/)

[
  "3.120.18.145/32",
  "3.120.73.242/32",
  "3.120.77.98/32",
  ...
]

You need the extractor to generate an array of objects and tell with the indicator which property in each object inside the extracted array contains the indicator.

Use the extrator "preview_cell_2.ip_ranges[].{indicator:@}" that will produce the following array of objects:

[
  {
    "indicator": "3.120.18.145/32"
  },
  {
    "indicator": "3.120.73.242/32"
  },
  {
    "indicator": "3.120.77.98/32"
  },
...
]

and use as "indicator" as indicator or just remove it as that is its default value.

Re: Polling JSON Format for Okta

doliver1 — Mon, 29 Oct 2018 16:19:11 GMT

@xhoms thanks for the catch on that. It works perfectly now.

Re: Polling JSON Format for Okta

KengSeng — Tue, 04 Dec 2018 08:44:58 GMT

Hi Xavier,

I tried to use this as extrator "*.ip_ranges[].{indicator:@}", but the AF minemeld doesnt allow me to click OK. I would like to extract everything from the Okta website.

Regards,

Keng Seng

Re: Polling JSON Format for Okta

KengSeng — Tue, 04 Dec 2018 09:19:44 GMT

I used the ubuntu VM is working.

age_out	default: null interval: 257 sudden_death: true
attributes	confidence: 100 share_level: green type: IPv4
extractor	*.ip_ranges[].{indicator:@}
indicator	indicator
url	https://s3.amazonaws.com/okta-ip-ranges/ip_ranges.json

Re: Polling JSON Format for Okta

xhoms — Tue, 04 Dec 2018 10:12:10 GMT

Hi @KengSeng,

AF-MM has a different UI and may have a different YAML validator. Try quoting the string ("*.ip_ranges[].{indicator:@}"). It should work with that.

Re: Polling JSON Format for Okta

KengSeng — Tue, 04 Dec 2018 10:52:17 GMT

Thanks Xavier, it works!!!

Re: Polling JSON Format for Okta

niuk — Tue, 17 Sep 2019 12:40:01 GMT

("*.ip_ranges[].{indicator:@}") doesnt pull any indicators for me , tried also (*.ip_ranges[].{indicator:@}), same thing I am using VERSION: 0.9.50 and 0.9.52