topic JSON parser extractor name with a "-" in General Topics

JSON parser extractor name with a "-"

MichaelPDean — Tue, 05 Feb 2019 20:39:42 GMT

I have a JSON list (URL https://ip-ranges.cloud.signiant.com/MediaShuttle) with one part of the JSON path name containing a minus sign ("-"):

{
    "us-east-1": {
        "all_ips": [
            "3.83.158.71", 
            "3.87.14.184", 
            "34.207.126.66", 
            "34.227.155.76", 
            "34.236.155.245", 
            "52.2.203.56", 
            "52.23.203.179", 
            "52.73.28.78", 
            "52.86.231.134"
        ]
    }
}

Using the JMESPath tester from jmespath.org, I created an expression that pulls out what I need:

us-east-1.all_ips

However, the Minemeld engine will not start, throwing an 'Unknown Character" error, apparently due to the dash. I've tried escaping and quoting using several variations and I cannot create the extractor needed. These are the variations I've tried:

us-east-1.all_ips: Unknown Character (the '-')
"us-east-1.all_ips": Unknown Character
"us-east-1".all_ips: Engine won't start
"us-east-1".all_ips[]: Engine won't start
'us-east-1'.all_ips: engine won't start
'us-east-1'.all_ips[].{indicator:@}: engine won't start
"us-east-1".all_ips[].{indicator:@}: engine won't start
us\-east\-1.all_ips[].{indicator:@}: Unknown character
"us-east-1.all_ips"[].{indicator:@}: engine won't start (expected end but found [)
"us-east-1.all_ips[]".{indicator:@}: expected <block end>, but found '<scalar>'
"us-east-1.all_ips[].{indicator:@}": Unknown character
"us\-east\-1.all_ips[].{indicator:@}": engine won't start

Thanks in advance,

Mike

Re: JSON parser extractor name with a "-"

xhoms — Wed, 06 Feb 2019 09:42:02 GMT

Hi @MichaelPDean,

this is a tricky one.

the right expression is already in your list (us-east-1.all_ips[].{indicator:@}) but it must be double escaped .

First, us-east-1 is not a valid python espression so you must quote it to become "us-east-1" . But you can't provide extractor: "us-east-1".all_ips[].{indicator:@} to the prototype because the YAML loader will fail parsing it due to the first double quote describing the value to be a string scalar and the dot character after the second double quote being considered as an invalid sequence.

So you must escape again for the YAML loader to consider the whole line a string scalar. The right value is then:

extractor: '"us-east-1".all_ips[].{indicator:@}'

Re: JSON parser extractor name with a "-"

MichaelPDean — Wed, 06 Feb 2019 16:01:53 GMT

Thanks, that worked!

Re: JSON parser extractor name with a "-"

joovoo — Tue, 25 Jan 2022 17:34:33 GMT

Thank you both. With this post I was able to decipher part of the syntax for a new miner. I can't seem to find a description of the syntax for the YAML files (as can be found in /opt/minemeld/prototypes/minemeld-node-prototypes/prototypes).