issues with concurrent api call

[antoinek]

Hello,

We have had a problem with concurrent API call for a long time and it gets worse as we have more equipment.
Sometimes if we have too much API call in the same time, we have errors: timeout, error 5xx or others strange erros.
The too much API call is realy really low, if we have 2 or 3 on the same time we have an issue

For example, in the traces below we have Session timed out and Invalid Credential when it is the same username for all the playbook tasks without any problems

Context:
- Modification of policy, NAT, users, migration of equipment or audit are done are performed automatically via API, particularly Ansible
- Panorama 10.2.4-h2
- 50 devices groups on this panorama and many ongoing migrations
- 250 commits per week so thousands or API call per weeks

The questions I have asked to Palo without responses:
- Are there any possibilities to speed up API call?
for example in Cisco ACI, we had change the auth from basic auth to cert and it's way better
- Is there some limits to the number of concurent API call?
- Is there a way to improve that?
- Is there some logs to analyse in more detail?

This is having a major impact because we have chosen PaloAlto as our main firewall supplier, but the automation is not keeping up and this is putting a question mark over future migrations.

I opened a ticket with TAC, who didn't see any error in the tech support and who disclaims all responsibility because it's Ansible...

Is anyone having limitation problems with the API?

Invocation

```
# Ansible Info
[root@f9cc47c8570d ~]# ansible --version
ansible [core 2.14.7]
config file = None
configured module search path = ['/root/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
ansible python module location = /usr/local/lib/python3.9/site-packages/ansible
ansible collection location = /root/.ansible/collections:/usr/share/ansible/collections
executable location = /usr/local/bin/ansible
python version = 3.9.16 (main, May 31 2023, 12:21:58) [GCC 8.5.0 20210514 (Red Hat 8.5.0-18)] (/usr/bin/python3.9)
jinja version = 3.1.2
libyaml = True

# Module collection
[root@f9cc47c8570d ~]# ansible-galaxy collection list | grep palo
paloaltonetworks.panos 2.13.3

# Python modules
[root@f9cc47c8570d ~]# python3.9 -m pip list | grep pan
pan-os-python 1.8.1
pan-python 0.22.0
pandevice 0.14.0
```

Traces

Examples of 2 concurent requests:

```
The full traceback is: File "/tmp/ansible_paloaltonetworks.panos.panos_address_object_payload_1ot7y5qi/ansible_paloaltonetworks.panos.panos_address_object_payload.zip/ansible_collections/paloaltonetworks/panos/plugins/module_utils/panos.py", line 250, in get_pandevice_parent
self.device = PanDevice.create_from_device(*pan_device_auth)
File "/usr/local/lib/python3.9/site-packages/panos/base.py", line 3748, in create_from_device
system_info = device.refresh_system_info()
File "/usr/local/lib/python3.9/site-packages/panos/base.py", line 4215, in refresh_system_info
system_info = self.show_system_info()
File "/usr/local/lib/python3.9/site-packages/panos/base.py", line 4171, in show_system_info
root = self.xapi.op(cmd="show system info", cmd_xml=True)
File "/usr/local/lib/python3.9/site-packages/panos/base.py", line 4000, in xapi
self._xapi_private = self.generate_xapi()
File "/usr/local/lib/python3.9/site-packages/panos/base.py", line 4072, in generate_xapi
"api_key": self.api_key,
File "/usr/local/lib/python3.9/site-packages/panos/base.py", line 3994, in api_key
self._api_key = self._retrieve_api_key()
File "/usr/local/lib/python3.9/site-packages/panos/base.py", line 4159, in _retrieve_api_key
xapi.keygen(retry_on_peer=False)
File "/usr/local/lib/python3.9/site-packages/panos/base.py", line 3897, in method
raise the_exception
failed: [oppano22s.intcs.meshcore.net] (item=bpfftc16s) => {
"ansible_loop_var": "item",
"changed": false,
"invocation": {
"module_args": {
"address_type": "ip-netmask",
"api_key": null,
"commit": null,
"description": null,
"device_group": "shared",
"gathered_filter": "name == bpfftc16s",
"ip_address": null,
"name": null,
"password": null,
"port": 443,
"provider": {
"api_key": null,
"ip_address": "oppano22s.intcs.meshcore.net",
"password": "VALUE_SPECIFIED_IN_NO_LOG_PARAMETER",
"port": 443,
"serial_number": null,
"username": "w120611"
},
"state": "gathered",
"tag": null,
"username": "admin",
"value": null,
"vsys": "vsys1"
}
},
"item": "bpfftc16s",
"msg": "Failed connection: URLError: code: 403 reason: Invalid Credential"
}
```

```
The full traceback is:
File "/tmp/ansible_paloaltonetworks.panos.panos_address_group_payload_71kgaaia/ansible_paloaltonetworks.panos.panos_address_group_payload.zip/ansible_collections/paloaltonetworks/panos/plugins/module_utils/panos.py", line 619, in apply_state
listing = obj.__class__.refreshall(obj.parent, add=False)
File "/usr/local/lib/python3.9/site-packages/panos/base.py", line 1326, in refreshall
raise e
File "/usr/local/lib/python3.9/site-packages/panos/base.py", line 1321, in refreshall
root = api_action(xpath, retry_on_peer=cls.HA_SYNC)
File "/usr/local/lib/python3.9/site-packages/panos/base.py", line 3897, in method
raise the_exception
failed: [oppano22s.intcs.meshcore.net] (item=bqfftc12s) => {
"ansible_loop_var": "item",
"changed": false,
"invocation": {
"module_args": {
"api_key": null,
"commit": null,
"description": null,
"device_group": "shared",
"dynamic_value": null,
"gathered_filter": "name == bqfftc12s",
"ip_address": null,
"name": null,
"password": null,
"port": 443,
"provider": {
"api_key": null,
"ip_address": "oppano22s.intcs.meshcore.net",
"password": "VALUE_SPECIFIED_IN_NO_LOG_PARAMETER",
"port": 443,
"serial_number": null,
"username": "w120611"
},
"state": "gathered",
"static_value": null,
"tag": null,
"username": "admin",
"vsys": "vsys1"
}
},
"item": "bqfftc12s",
"msg": "Failed gathered_filter refresh: Session timed out"
}

```

Best regards,

Antoine,

[antoinek]

So after many exchanges with support, many tries and debugging, here is what I learned about the concurent API call:

They is something on the documentation (https://docs.paloaltonetworks.com/content/dam/techdocs/en_US/pdf/pan-os/10-1/pan-os-panorama-api/pan...) and the limitation is concurent request to five when login:
"
When multiple login or logout events are generated at the same time, make sure to
follow these guidelines to ensure optimal firewall performance:
• Design your application to queue events and perform batch API updates instead of
sending single event or mapping updates.
• Limit the number of concurrent API calls to five. This limit ensures that there is no
performance impact to the firewall web interface as the management plane web
server handles requests from both the API and the web interface.

"
This is not a guidelines but an hard limit where "there no option to exceed the limits".

Also this limit is not only for login "- Is this limit only for login? NO"

It's not 5 per seconds but 5 concurent (I prefer to clarify because it was the TAC that was giving us 5 per second).
I managed to go to more than 5 API request per second without any problems.

Our solution was to generate the API key once and us it for authentication every time instead of the user's username and password, as this generated a lot of requests and load on the panorama.

For the moment this solves our problem or we were waiting for the limit, to be followed in the future.
But we hope that such a low hardcoded limit can be modified.

It's a bit annoying that the documentation isn't clearer on this point, and it didn't seem very clear to the TACs either.


Antoine