topic Re: Wildcards in rest api configuration query? in Automation/API Discussions

Wildcards in rest api configuration query?

KostiM — Tue, 28 Mar 2017 07:11:58 GMT

Hi,

I'm trying to write a reporting tool that will go throguh various device groups and identify rules that have certain keywords (change numbers in my case) in the description.

I'm querying panorama configuration and using rest api directly from python (e.g. requests module or pan.xapi)

For example, this query works perfectly for me and as a result I get two rules returned back, if description contains only one string CHG99646:

/config/devices/entry[@name='localhost.localdomain']/device-group/entry[@name='Perth']/pre-rulebase/security/rules/entry[description="CHG99646"]

<entry name="Rule139" loc="Perth">...</entry>
 ...
 
 
However, my queries stop working in case description contains multiple lines. E.g if my rule entry is like this:
 
<response status="success" code="19">
<result total-count="1" count="1">
<entry name="Rule87" loc="Perth">
<profile-setting>
<group>
<member>Source123</member>
</group>
...
...
<action>allow</action>
<description>CHG128742 CHG135069 CHG139311 CHG142080</description>
 
I am unable to find this rule if I put a query like
/config/devices/entry[@name='localhost.localdomain']/device-group/entry[@name='Perth']/pre-rulebase/security/rules/entry[description="CHG139311"]
 
Is there any way to have ~ (like) matching here or to use wildcards (e.g. .*<matchpattern>.*) so I could identify rule(s) based on part of the description content and not the entire decription content?
 
I am aware that I can pull the entire rulebase and then parse it localy. But the point here is to avoid this. If there are e.g. 20 device groups and each device group has 1000+ rules, pulling 20.000 rules will take forever and it would be much more efficient if I can just return rules matching certain patterns like explained above.
 
Thanks,
Milos

Re: Wildcards in rest api configuration query?

cpainchaud — Tue, 28 Mar 2017 07:20:39 GMT

Hi,

The API is not designed for that. It has limited support for XPath and pulling data locally is probably the only way.

There are as well exisiting utilities which can do what you are looking for already, like one of mine here: https://live.paloaltonetworks.com/t5/API-Articles/rules-edit-php-to-manage-edit-export-rules-from-CLI/ta-p/53321

Re: Wildcards in rest api configuration query?

KostiM — Tue, 28 Mar 2017 07:46:45 GMT

OK, so Palo Alto decided not to implement %like% in their rest api. Understood.

Do you know if there is an op command that I could incorporate instead of using wildcards in the xpath matching? Something similar to global find in panorama gui or a | match when I am running show configuration? I mean, anything that will return a security rule name(s) based on text I want to find in description.

Cheers,

Milos

Re: Wildcards in rest api configuration query?

cpainchaud — Tue, 28 Mar 2017 07:50:58 GMT

What I mean is that PAN is using Xpath, so you can use what Xpath has available:

In your case, something like that should work :

[contains(text(),'whatYouWantHere')]

concerning your other question about Global Find and such : it's not exposed to through API unfortunatly. But it's somethign available on my project if you dont mind using it and/or developping in PHP 😉

Re: Wildcards in rest api configuration query?

KostiM — Tue, 28 Mar 2017 07:56:26 GMT

I'm trying to use your suggestion with contains text but I am not able to match anything

Could you give a full xpath example for matching description field with [contains(text(),'whatYouWantHere')]? I'm missing something when trying to use your suggestion.

For the PHP I have minimal knowledge there 🙂

Milos

Re: Wildcards in rest api configuration query?

KostiM — Tue, 28 Mar 2017 08:06:24 GMT

YES! Figured it out:

/config/devices/entry[@name='localhost.localdomain']/device-group/entry[@name='Perth']/pre-rulebase/security/rules/entry[description[contains(text(), 'CHG129149')]]

This works as a charm and gives me exactly what I need.

Thsi is so cool. I am going to accept your answer as the solution!

Thanks so much for your help.

Milos

Re: Wildcards in rest api configuration query?

cpainchaud — Tue, 28 Mar 2017 08:27:30 GMT

you are welcome! Remember it's just standard XPath 1.0 so you can google for more examples of syntaxes

Concerning my PHP lib: it's a set of utilities that are ready to run (no scripting skills involved, just invoke them) and a framework for devs. You might want to have a look if you dont want to reinvent the wheel.

cheers !

Re: Wildcards in rest api configuration query?

KostiM — Thu, 30 Mar 2017 03:52:49 GMT

I see that you are very good with xpath 1.0. I'm wondering if you have any suggestions/solutions for the following scenario or if this is not possible to achieve.

Say I use query xpath like this:

/config/devices/entry[@name='localhost.localdomain']/device-group/entry[@name='Tokyo']/pre-rulebase/security/rules/entry[description[contains(text(), 'CHG84226')]]

As a response I get the entry (or entries) with attribute 'loc' that's set to the current device group:

<profile-setting>

<group>

<member>SecProfile2</member>

</group>

</profile-setting>

Now if I change Xpath to ommit the part for device group as I want to build a search that will identify description with certain CHG text value on all device groups and find security rules accross all device groups, I get the positive response and entry is listed back, but I lose the *loc* attribute so my policy doesn't show where exactly it belongs to.

Example (device group entry part is replaced with //:

/config/devices/entry[@name='localhost.localdomain']/device-group//pre-rulebase/security/rules/entry[description[contains(text(), 'CHG84226')]]

<disable-server-response-inspection>no</disable-server-response-inspection>

</option>

<from>

<member>trust</member>

</from>

<to>

<member>untrust</member>

</to>

Any ideas what could I possibly use in xpath to return something meaningful that can help me identifying where exactly this rule belongs to (rules in the case above as 'CHG84226' exists as a description on 8 rules accross 8 different device groups). I know that I can use rule name and query individual device groups one by one but in my case I've got 20 device groups and one query costs around 1 second so it adds up to a large waiting time when the script is executed.

If you reckon I should post this as a separate question, I don't mind reposting this in its entirety.

Thanks for suggestions.

Re: Wildcards in rest api configuration query?

cpainchaud — Thu, 30 Mar 2017 08:38:45 GMT

this 'loc' property is added by PANOS, it's not part of XPath at all.

The fact it displaying in one case but not another is a mystery to me : it's not even a 'documented' feature of PANOS.

You will find yourself quickly limited with XPath because it doesn't fit all purposes.

I would usually script that : download the candidate config -> parse what I want (you can use XPath inside python as well if you don't want to code navigate through Xml tree by yourself.

Can I ask what kind of action you would like to do after you foudn the rules you need ?

Re: Wildcards in rest api configuration query?

KostiM — Thu, 30 Mar 2017 08:51:40 GMT

Hello again,

I'm writing an application in Python Django to expose certain areas of palo altos to our developers so they can be self sufficient and be able to identify changes that we made on firewalls, find out if their application is being blocked by deny policy and so on. We don't want to give people access to palo altos themselves + most of the people wouldn't know what to do once they connect to firewalls.

In my way, my simple web interface would provide just enough functionality to do simple tasks such as what I mentioned above.

I get your point for downloading the entire config. In my case that won't be efficient.

$ ./rt.py

Start

/config/devices

End request, returned XML in 28.5984241962

Converting XML to DICT

End xmltodict

Done

Finished conversion 44.2386090755

It takes ~28 seconds to download the entire config in XML format. As I can't really use XML for the rest of my python stuff and Dictionary / OrderedDict is something much more useful, I then convert XML to Python Dictionary. As you can see it takes additional 16 seconds to convert literary everything to a massive dict that I'm only using a ridiculously small portion of.

In total 44.2 seconds, that's a lot of time to waitfor each request 😉

And all I need to do is to display few policies to someone who enters a description text to search for. I don't think that's very efficient and this is why I am trying to find the way to limit the output as much as I can with xpath but still have enough information to work with to display necessary info to the user.

Cheers,

Milos

Re: Wildcards in rest api configuration query?

cpainchaud — Thu, 30 Mar 2017 09:01:29 GMT

Unfortinately, PANOS doesn't provide API features to search for objects and such. The XPath is oringally designed to name a 'path' in the XML rather than doing searches.

Why does it take so long to get the condicate config ? Bandwidth issue or PANOS is too slow ? It should be instant almost

Re: Wildcards in rest api configuration query?

KostiM — Thu, 30 Mar 2017 09:08:08 GMT

Config is 18MB in size

$ ./rt.py

Start

/config/devices

End request, returned XML in 28.5243420601

Size in bytes: 18714100

Converting XML to DICT

End xmltodict

Done

Finished conversion 44.2015681267

And palo alto Panorama is located on the other side of the world (accessing it over WAN)

What is your config size that you are normally downloading when you say it is almost instant?

Even if it was instant I still have about 18 seconds to convert XML to Dictionary but I could live with that (if only was instant as you say 🙂 🙂 )

Re: Wildcards in rest api configuration query?

cpainchaud — Thu, 30 Mar 2017 09:14:42 GMT

I quit using XML to dictionary/json/arrays a long time ago : it's way too slow. I parse XML directly with libxml and alike.

In your case, to solve your bandwidth issues, I would have a middleware/application-proxy that would be excuted in the same datacenter as the firewall/panorama. This mini-API of yours would be able to acess the firewall without any bandwidth or latency constraint and would then just return the pieces of interest to you.

Re: Wildcards in rest api configuration query?

cpainchaud — Thu, 30 Mar 2017 09:17:39 GMT

client requests MiddleAPI.getAllRulesMatchingX ->

MiddleAPI requests PAN_API.getFullConfigDump

MiddleAPI parse XML, filters records of interest, add a 'loc' property

MiddleAPI sends answer to client