Purpose of this document
This document is being prepared to capture best practices and recommendations for Panorama configuration and usage for scaled deployments in order to get an optimized performance in terms of UX and commit times. This document includes some of the old best practices that are already documented here, and some new ones.
Design
Configuration Size
All the configuration on Panorama, including configuration on managed devices as well as on Panorama itself, is stored in an XML file. The size of this file is referred to as Panorama Configuration Size.
On an average, we can say that 1000 Policies (Security, NAT, etc) can contribute approximately 2MB to the config size, while 12000 Objects (Addresses, Address Groups, etc) can contribute approximately 2MB to the config size. Please note that the above calculation can vary depending on the associated configuration like description, member addresses, etc.
Device Groups
- Consider cleaning up Device Groups that are not associated with any devices and do not have any child Device Groups.
- In the Shared device group, keep only the policies and objects that you intend to share across all the managed firewalls. Rather, leverage the device group hierarchy to share configuration only across the relevant devices.
Managing configuration objects at the appropriate device group level helps minimize the number of Out of Sync firewalls more efficiently because the configuration of all the firewalls become Out of Sync if a single shared configuration object is modified.
This also drastically improves Commit times because of the reduced number of validations needed.
- Associate Reference Templates to refer to network configuration objects contained in a template that the managed firewall does not belong to in order to complete a security configuration.
- This allows you to take full advantage of common configuration objects across device groups and templates without overuse of the Shared device group or recreating the identical network configuration objects.
- For an optimal security posture, consider leveraging the rule evaluation order while defining your policies.
For example, create any Security pre-rules that you want managed firewalls to apply without exception while creating Security post-rules to act as a cleanup for any traffic that did not match a Security pre-rule.
- Consider following a configuration optimization (removing unused policies and objects) cycle at regular intervals.
Templates, Stacks and Variables
- Go modular by creating templates with logical groupings of settings even if the configuration is incomplete.
Remember, the configuration must be complete and all references resolved at the template stack level—not at every template. You can reuse, reference, and override objects from different templates to complete the template stack configuration.
- Avoid overriding template stack configuration.
Alternatively, you can leverage the stack capabilities to override the values using templates within the stack. You could also leverage template variables to configure different values for different firewalls within the same template stack.
- While configuring template variables, assign a default value of None to ensure that incorrect configuration is not accidentally pushed to the managed firewalls.
- If you’re using Selective Push, do not use CSV Import to override Template Stack Variables. Instead, you can either override the variable values manually at the Template Stack level, or at the Device level under Panorama > Managed Devices > Summary.
Commit and Push operations
Factors affecting Commit times
- Number of Shared Objects or Policies in Total Configuration
Every time a change is made to a Shared configuration, every Device Group that shares the config will be reevaluated and regenerated in the Commit cycle. This adds to overall Commit time.
- Number of uncommitted changes at the time of Commit
The commit time increases drastically when the number of uncommitted changes are higher than 250.
An uncommitted change refers to changes that are in the candidate config, but are not included in the current Commit scope. Uncommitted changes are usually seen in cases like;
- multiple admins operating on Panorama concurrently and attempt to commit their changes alone.
- admins performing partial commit of changes.
- Static Address Objects are present in Dynamic Address Groups
If tags used in filter criteria of DAGs match static objects then it impacts the configuration size and push operation completion time.
- Plugins installed and in use
Plugin operations come with their own set of validations which are added into the Commit cycle, again adding to the overall Commit time.
- Frequency of Commits
Panorama has a Commit Queue size of 10, which means if there are more commits being queued than can be processed, we will start experiencing slow commit times.
For example, if there are many admins making changes on Panorama and committing at the same time, we might hit the Commit Queue limit of 10, thus blocking admins from adding any further commits until the queue is cleared up.
If there are multiple config changes by different admins that require commit, follow a commit cycle window where all admins make their changes and a full commit is done at regular intervals. A full commit will push all the pending changes by all the admins as a single commit. Following this will avoid hitting the limit of 10 in the commit queue.
- Total Configuration Size
A large Configuration size means packing, unpacking and repacking a huge XML file every time there is a change, which inadvertently contributes to prolonged Commit times.
Factors affecting Push times
Push operations on Panorama basically translate to the below operations;
- Transferring the transformed config to the managed device
Basically, the configuration set intended for the target firewall will be extracted from the corresponding Device Group and Template Stack and be transformed to the configuration set matching the Software version on the target firewall.
- Committing the new config on the managed device
The factors that affect Push times on Panorama are;
- The Share Unused Objects setting
When this option is disabled, Validations include processing of Objects to identify Unused Objects to avoid pushing them to the target firewalls, thereby adding to overall Commit time while pushing config to firewalls.
- Number of Template Variables
Using a large number of template variables within the templates in a stack also require further post-commit processing, thereby impacting push times.
- Policy Exclusivity within Device Groups
If Policies are configured to apply only to specific targets within a Device Group, Post-Commit processing is required to ensure that the reference validations are performed correctly. This also results in delayed Push times.
- Different Software Versions on Panorama and Firewalls
When the Firewall is on a software version that is much lower than the version on Panorama, Panorama has to transform the configuration to support the lower version on the Firewall every time there is a push.
- Too many unpushed changes
While using Selective Push, having too many unpushed changes leads to UI sluggishness when loading the Push scope window, also impacting Push times.
- Enabling Shared to Shared optimization
Enabling Shared to Shared optimizations ensures that Shared vsys configurations are not copied across the multiple vsys on the firewall. However, this also requires extra processing and validations on Panorama at Push times.
- Older Firewall models
Firewall models like the old 800 series have much longer Commit times.
- Large Config change being pushed from Panorama to Firewall
When the change being pushed to the firewall is large, there can be small delays in the config relay to the firewall as well as prolonged commit times on the firewall itself
- Latency between Panorama and managed device
Higher latency between Panorama and the managed firewall can further contribute to longer push times.
- Disconnected Managed Devices
If disconnected devices are included as Targets in the Push scope, it would lead to longer push times.
Best Practices for Commit and Push operations
- As best possible, ensure that the managed firewalls are on the same PAN-OS release train as Panorama. This ensures a reduced or no effort on Panorama to transform the configuration to the lower firewall version.
For example, if Panorama is at 11.1.x, it is recommended to keep the managed firewalls also in the 11.1.x versions.
- Avoid including too many changes (more than 100 config changes) in a local single commit during peak hours of Panorama usage. Instead, you can split the commits into smaller changes and commit accordingly.
- When multiple users are committing changes, review the changes in the Commit scope to ensure that they are not including changes from other users. Also, verify that target firewall list in the Push scope before pushing the changes from Panorama.
- Avoid having too many uncommitted changes in the candidate configuration while performing Selective Commit and Push. It is recommended to keep the number of uncommitted changes to less than 250.
- If Selective Push is used quite a lot, consider performing a Full Push at periodic intervals to ensure that the devices are In Sync with the configuration on Panorama.
- Commit operations consisting of Load, Revert and Bulk Delete config operations that include a lot of changes are heavy operations. Avoid these operations during hours of peak usage and perform full commit with these operations.
- At the time of a software upgrade (or downgrade), it is recommended that you do a Full Push to remove all uncommitted changes from the configuration.
Concurrent Admins and API usage
Almost every operation performed on Panorama is through an API request, including logged-in users, XMLAPI and RESTAPI calls using automation and also APIs called by third party vendors like Algosec or Tufin. Too many concurrent API requests can result in slowness in Panorama performance.
For performance considerations, limit the number of concurrent API calls to five. The suggested limit is to avoid the performance impact to the Panorama web interface as the management plane web server handles requests from both the API and the web interface. Limits may vary depending on the type of request. The limit may be higher depending on requests.
Factors affecting UI and API performance
- Too many UI operations in quick succession
If the UI is already slow in its responses, too many UI operations may further deteriorate the UI performance. For example, moving between Tabs, multiple clicks, multiple tabs, etc.
- Too many users operating concurrently
If there are multiple users logged in and operating on the Panorama appliance at the same time, it results in slow response times on the UI.
- Heavy API usage through automation or 3rd party tools
Use of 3rd party tools to monitor Panorama or managed firewalls through Panorama, can also cause resource exhaustion on the Panorama appliance, leading to sluggish performance on the UI as well. A lot of times, the 3rd party tools could be frequently making a single change and performing a commit.
- Config Locks:
To avoid config errors due to concurrent admin activities, the system allows for one config change operation at a time while config fetch operations can be run in parallel. Timing for each config change operation depends on the structure and layout of configuration elements, for example write/read operations for shared objects would require longer time to process at the backend.
For example, If admin1 is performing a heavy operation such as load partial or revert config change on a Panorama with 150MB config size, this operation will require a long time to process the entire config and make changes, especially if it is for shared objects. Meanwhile if admin2 & admin3 are trying to execute a write operation that requires the backend database to be edited, these admins will experience an auto config lock or delay in UI response.
- Compute-intensive operations
Operations that modify the XML schema like Load, Revert and Delete can be very compute-intensive and can lead to the UI being sluggish.
- Plugins
Sometimes, plugins like the Cloud Services plugin may also require some validations on the UI on some operations. In those cases, the UI responses will be impacted.
Best practices for an optimum UI and API performance
- In case of Panorama HA deployment, use the Passive Panorama as the target server for any Monitoring operations to reduce the API load on the Active Panorama.
- Reduce the frequency of API calls for monitoring via 3rd party tools. See if the 3rd party tools are making recursive calls for single config change and consecutive commits. This will impact the UI performance and experience for admins logged in via UI.
- During peak hours of usage, APIs meant for firewalls should not be redirected to Panoramas.
- Avoid multiple clicks on the UI in cases where the UI is not responding. Instead, wait for a response from the initial operation before proceeding to the next.
Conclusion
With the information provided here, we can try to optimize panorama performance and be aware of the most common factors affecting panorama performance. Looking at these factors can help isolate the root cause quicker and adopting the best practices will result in an improved system performance.
