pandevice newbie issues with fetching objects (addressobject, addressgroup, serviceobject...

[Arkadiusz_Majer]

Hello,

I'm not pretending to be a pandevice guru... moreover, I'm not pretending to be python guru. Rather basic skills in both. I've got a task to retrieve firewall configuration (rules + object definitions) from two datacenters (primary & DR, separate setups), store this in some reasonable format and compare to find differences and make sure that they are in sync (or at least to make sure that differences are related to different IP scheme and some minor functionality differences - in DR we have some additional test environment).

High level setup is that in both DC we have Panoramas and PA firewalls managed by those Panoramas. We use Device-Groups in Panoramas and several vsys in PA FW. We use both physical PA FW and virtual appliances (private cloud). Device-Groups include at least 3 levels of inheritance (Shared > DataCenter > areaX, Shared > privcloud > areaY and similar)

We decided that using pandevice might help us with fetching required information and store rule&object definitions in JSON files then use some handcrafted tool to make 'smart diff' (checking some basic rule details, like focusing on source/dest/source zone/dest zone/service). For some reasons Python 2.7 is being used. However, using 3.7 does not make any difference (tried).

I'm able to fetch prerulebase and postrulebase, connecting to all interesting Device-Groups. I can build a complete ruleset for each firewall based on those rulebases and checking target devices included in rules. But I'm not able to fetch any addressobject/addressgroup or service objects. We are creating objects mostly in DataCenter device group (see scheme above) and I see them in Panorama or by looking at config via CLI, but when I try to fetch them by using pandevice I get 0.

I'm using this to fetch rulebases (working, Py27):

from __future__ import print_function, division
import sys
import io
import json
import pandevice
# from pandevice import base
# from pandevice import firewall
from pandevice import panorama
from pandevice import objects
from pandevice import policies

DEVICES = ["a.b.c.d", "e.f.g.h"]   # one Panorama in main DC and the other in DR DC
DEVICEGROUPS = ["Shared", "DataCenter", "areaX", "areaY"]   # Shared > DataCenter > areaX, Shared > DataCenter > areaY
PAN_API_KEY = "xxxxxxxapikeyxxxxx"

for device in DEVICES:
    pano = panorama.Panorama(device, api_key=PAN_API_KEY)
    for devgrp in DEVICEGROUPS:
        panogrp = panorama.DeviceGroup(devgrp)
        pano.add(panogrp)

        # PreRulebase
        # Build the object tree
        prb = policies.PreRulebase()
        panogrp.add(prb)
        # Fetch rules
        all_rules = policies.SecurityRule.refreshall(prb)
        if all_rules:
            combined_list = []
            iteration = 1
            for element in all_rules:
                # iteration - starting with 1, not 0
                # Populate dictionary mappings
                i = {
                    '_seq_': iteration,
                    '_name_': element.name,
                    'fromzone': element.fromzone,
                    'tozone': element.tozone,
                    'source': element.source,
                    'source_user': element.source_user,
                    'destination': element.destination,
                    'application': element.application,
                    'service': element.service,
                    'category': element.category,
                    # ... etc, enumerated fields that are interesting for our analysis
                }
                iteration += 1
                combined_list.append(i)
            # dumping ALL rules to JSON file, make sure that we use correct UTF-8 encoding and proper
            # line-endings (LF only, unix-like)
            with io.open("json/" + device + "_" + devgrp + "_prerulebase_processed.json",
                         "w", newline="\n", encoding="utf8") as outfile:
                data = json.dumps(combined_list, sort_keys=True, indent=4, separators=(',', ': '),
                                  ensure_ascii=False)
                outfile.write(unicode(data))
                outfile.write(unicode('\n'))

        # PostRulebase
        # Build the object tree
        prb = policies.PostRulebase()
        panogrp.add(prb)
        # Fetch rules
        all_rules = policies.SecurityRule.refreshall(prb)
        if all_rules:
                  #
                  # the same approach as in PreRulebase - get fields, output to JSON
                  #

After running above code I'm getting several JSON files with prerulebases and postrulebases for each defined Device Group, for each of two listed Panoramas. Actually this code comes from various pandevice examples and from some discussions here in this forum.

Now, similar approach for address/service objects:

...
    #
    # Get Objects for each listed device group
    for devgrp in DEVICEGROUPS:
        panogrp = panorama.DeviceGroup(devgrp)
        pano.add(panogrp)
        #
        # AddressObjects
        aobjects = pandevice.objects.AddressObject()
        pano.add(aobjects)
        current_object_list = pandevice.objects.AddressObject.refreshall(pano)
        # (tried also pandevice.objects.AddressGroup.refreshall(aobjects)
        objects_list = []
        for object_element in current_object_list:
            obj = {
                'name': object_element.name,
                'value': object_element.value,
                'type': object_element.type,
                'description': object_element.description,
                'tag': object_element.tag
            }
            objects_list.append(obj)
       if objects_list:
          with io.open("json/" + device + "_" + devgrp + "_addressobjects.json",
                       "w", newline="\n", encoding="utf8") as outfile:
              data = json.dumps(objects_list, sort_keys=True, indent=4, separators=(',', ': '),
                            ensure_ascii=False)
              outfile.write(unicode(data))
              outfile.write(unicode('\n'))

        #
        # AddressGroups
        agobjects = pandevice.objects.AddressGroup()
        pano.add(agobjects)
        current_object_list = pandevice.objects.AddressGroup.refreshall(pano)
        # (tried also pandevice.objects.AddressGroup.refreshall(agobjects)
        objects_list = []
        for object_element in current_object_list:
            obj = {
                'static_value': object_element.static_value,
                'dynamic_value': object_element.dynamic_value,
                'description': object_element.description,
                'tag': object_element.tag
            }
            objects_list.append(obj)
        if objects_list:
             with io.open("json/" + device + "_" + devgrp + "_addressgroups.json",
                          "w", newline="\n", encoding="utf8") as outfile:
                 data = json.dumps(objects_list, sort_keys=True, indent=4, separators=(',', ': '),
                               ensure_ascii=False)
                 outfile.write(unicode(data))
                 outfile.write(unicode('\n'))

I've tried several combinations when refreshing objects but with no luck. Everytime I'm just getting nothing, empty lists.

I probably do something obviously wrong but I have no experience to find out what.

Any hints? Any help here?

Thanks!

Arek

[gfreeman · Accepted solution]

The problem is your pandevice object hierarchy tree.  When you are getting security rules, your tree needs to look like this:

Panorama > device group > rulebase > (security rules)

For address objects, it should be this:

Panorama > device group > (address objects)

So in your second script, you're invoking pandevice.objects.AddressGroup.refreshall(aobjects), but you should using the device group devgrp as the param passed in to the address objects/group's refreshall().

If you look at the CHILDTYPES for each object, you'll see which objects can be children of that object, so that is the way to see what goes with what.

One other final tweak, you do not need to manually turn pandevice objects into JSON, there is an .about() every object has that you should use which will give you a dict of the params.

[Arkadiusz_Majer]

Thank you so much!

Changing 'scope' helped.

for device in DEVICES:
    pano = panorama.Panorama(device, api_key=PAN_API_KEY)

    #
    # when you need to fetch SHARED objects (common to all Device Groups), like AddressObject, AddressGroup, ServiceObject, ServiceGroup
    current_object_list = pandevice.objects.AddressObject.refreshall(pano)
    objects_list = []
    for object_element in current_object_list:
        obj = {
            'name': object_element.name,
            'value': object_element.value,
            'type': object_element.type,
            'description': object_element.description,
            'tag': object_element.tag
        }
        objects_list.append(obj)

    #
    # when you need to fetch objects in the Device Group
    for devgrp in DEVICEGROUPS:
        panogrp = panorama.DeviceGroup(devgrp)
        pano.add(panogrp)
        current_object_list = pandevice.objects.AddressObject.refreshall(panogrp)
        objects_list = []
        for object_element in current_object_list:
            obj = {
                'name': object_element.name,
                'value': object_element.value,
                'type': object_element.type,
                'description': object_element.description,
                'tag': object_element.tag
            }
            objects_list.append(obj)

Now I'm able to retrieve interesting objects and build a complete rulesets.

If I'm not mistaken, the same rule is for rulesets - common prerulebase/postrulebase (ie, seen as Shared). We don't have any 'Shared' rules so I can't confirm that.

Thanks! 🙂

[Arkadiusz_Majer]

btw

I'm not quite sure what do you mean by using "about()". Is just about getting schema for objects? or, can I use it to dump list of objects?

Something like this?

        current_object_list = pandevice.objects.AddressGroup.refreshall(panogrp)
        objects_list = []
        for object_element in current_object_list:
            objects_list.append(object_element.about())

[TeckSeng]

Question - to add a new address object, do we need to download the entire object list then append the new object before uploading it back to the firewall or panarama ? Or just add the new object only ? Thank you.

[Datafield]

Hi guys, I used the same format above. I am trying to pull objects from my rule but the interesting objects are in csv. The code is running with no errors but I dont get any output. Please let me know what might be missing . This is my code:

from __future__ import print_function, division
import sys
import io
import json
import panos
from panos import panorama
from panos import objects
from panos import policies
import csv

password = "Kunle@1984"
username = "admin"
pan_ip = "10.10.200.145"
pano = panorama.Panorama(pan_ip, username, password)
rule_file = "lab9-rules.csv"

file_ = open(rule_file, 'r')
csv_reader = csv.reader(file_)

for row in csv_reader:

value = row[0]
prb = policies.PreRulebase()
pano.add(prb)
# Fetch rules
all_rules = policies.SecurityRule.refreshall(prb)
if all_rules:
combined_list = []
iteration = 1
for element in all_rules:
# iteration - starting with 1, not 0
# Populate dictionary mappings
i = {
'_seq_': iteration,
'_name_': element.name,
'fromzone': element.fromzone,
'tozone': element.tozone,
'source': value,
'source_user': element.source_user,
'destination': element.destination,
'application': element.application,
'service': element.service,
'category': element.category,
# ... etc, enumerated fields that are interesting for our analysis
}
iteration += 1
combined_list.append(i)
# dumping ALL rules to JSON file, make sure that we use correct UTF-8 encoding and proper
# line-endings (LF only, unix-like)
with io.open("json/" + row + "_" + "_prerulebase_processed.json",
"w", newline="\n", encoding="utf8") as outfile:
data = json.dumps(combined_list, sort_keys=True, indent=4, separators=(',', ': '),
ensure_ascii=False)
outfile.write(unicode(data))
outfile.write(unicode('\n'))