Polling JSON Format for Okta

cancel
Showing results for 
Search instead for 
Did you mean: 

Polling JSON Format for Okta

L0 Member

I am trying to create a prototype for a Miner that pulls IP's from a JSON formatted file. I have looked at the documentation for setting up a JSON miner (https://live.paloaltonetworks.com/t5/MineMeld-Articles/Using-MineMeld-to-extract-indicators-from-a-g...) and I am having little luck as the error messages in the logs are saying there is a parsing error.

 

File "/opt/minemeld/engine/0.9.48.post1/local/lib/python2.7/site-packages/minemeld/ft/basepoller.py", line 606, in _polling_loop
ipairs = process_item(item)
File "/opt/minemeld/engine/0.9.48.post1/local/lib/python2.7/site-packages/minemeld/ft/json.py", line 192, in _process_item
indicator = item[self.indicator]
TypeError: string indices must be integers

 

My Config looks like the following:

age_out:
default: null
interval: 257
sudden_death: true
attributes:
confidence: 100
share_level: green
type: IPv4
extractor: preview_cell_2
indicator: ip_ranges
source_name: aws.S3
url: https://s3.amazonaws.com/okta-ip-ranges/ip_ranges.json

 

Can anyone assist?

1 ACCEPTED SOLUTION

Accepted Solutions

L5 Sessionator

Hi @doliver1,

 

your extractor/indicator combination produces an array of strings instead of an array of objects (you can verify it at http://jmespath.org/)

 

[
  "3.120.18.145/32",
  "3.120.73.242/32",
  "3.120.77.98/32",
  ...
]

You need the extractor to generate an array of objects and tell with the indicator which property in each object inside the extracted array contains the indicator.

 

Use the extrator "preview_cell_2.ip_ranges[].{indicator:@}" that will produce the following array of objects:

 

[
  {
    "indicator": "3.120.18.145/32"
  },
  {
    "indicator": "3.120.73.242/32"
  },
  {
    "indicator": "3.120.77.98/32"
  },
...
]

and use as "indicator" as indicator or just remove it as that is its default value.

View solution in original post

7 REPLIES 7

L5 Sessionator

Hi @doliver1,

 

your extractor/indicator combination produces an array of strings instead of an array of objects (you can verify it at http://jmespath.org/)

 

[
  "3.120.18.145/32",
  "3.120.73.242/32",
  "3.120.77.98/32",
  ...
]

You need the extractor to generate an array of objects and tell with the indicator which property in each object inside the extracted array contains the indicator.

 

Use the extrator "preview_cell_2.ip_ranges[].{indicator:@}" that will produce the following array of objects:

 

[
  {
    "indicator": "3.120.18.145/32"
  },
  {
    "indicator": "3.120.73.242/32"
  },
  {
    "indicator": "3.120.77.98/32"
  },
...
]

and use as "indicator" as indicator or just remove it as that is its default value.

View solution in original post

@xhoms thanks for the catch on that. It works perfectly now.

Hi Xavier,

 

I tried to use this as extrator "*.ip_ranges[].{indicator:@}", but the AF minemeld doesnt allow me to click OK. I would like to extract everything from the Okta website.

 

Regards,

Keng Seng

I used the ubuntu VM is working.

 

age_out
  • default: null
  • interval: 257
  • sudden_death: true
attributes
  • confidence: 100
  • share_level: green
  • type: IPv4
extractor *.ip_ranges[].{indicator:@}
indicator indicator
url https://s3.amazonaws.com/okta-ip-ranges/ip_ranges.json

Hi @KengSeng,

 

AF-MM has a different UI and may have a different YAML validator. Try quoting the string ("*.ip_ranges[].{indicator:@}"). It should work with that.

Thanks Xavier, it works!!!

 ("*.ip_ranges[].{indicator:@}") doesnt pull any indicators for me , tried also  (*.ip_ranges[].{indicator:@}), same thing I am using VERSION: 0.9.50  and 0.9.52

Like what you see?

Show your appreciation!

Click Like if a post is helpful to you or if you just want to show your support.

Click Accept as Solution to acknowledge that the answer to your question has been provided.

The button appears next to the replies on topics you’ve started. The member who gave the solution and all future visitors to this topic will appreciate it!

These simple actions take just seconds of your time, but go a long way in showing appreciation for community members and the LIVEcommunity as a whole!

The LIVEcommunity thanks you for your participation!