Intermittent 403 - Failed Connection Errors in Ansible Playbook

[Siddhant]

I have an ansible playbook that creates address,service objects -> security policy -> Commit and push to different device groups. 
Randomly one of the task fails during executing with the error - Failed Connection: URL Error: code: 403 reason: Forbidden. 
This is not specific to any particular module and have seen it happening in panos_address_object, panos_commit_push etc. Any guidance on this ? 

[nikoolayy1]

You are using the REST-API right ? If so maybe you have generated an API key from a username that is not an admin with full permisions ? Because you mention device groups I think that you are using the Ansible with an API key to control Panorama and the error 403 also confirms that REST-API is used not ssh. It is possible that your user that you use the API_key in the Ansible may have access domain just to some device groups or templates.

https://docs.paloaltonetworks.com/pan-os/9-0/pan-os-panorama-api/get-started-with-the-pan-os-xml-api...

https://docs.paloaltonetworks.com/panorama/9-0/panorama-admin/panorama-overview/role-based-access-co...

[Siddhant]

Thanks for your response ! 🙂 I am using the panos ansible modules to run these tasks against the panorama. 
In our case, the service account user has admin access (it is a superuser) to all the templates, device groups and we are using api_username and api_password to authenticate to the device.

Another interesting fact is it works while running most of tasks in the playbook , but randomly fails on one of them (And sometimes it doesn't fail). The panorama logs indicate that "Authorization Failed. Could not find the role/ado for the user <service_account>. However after checking the Remote auth server logs and policies looks like the policies and roles have been configured correctly on the Auth server. 

Do you have any other suggestions ? Thanks a lot in advance for your help ! 

Regards
Siddhant Kulkarni

[abedJawhar]

We have the same issue when calling some API endpoints.. It happens randomly and once we retry the same exact call with the same exact parameters, it works fine.. We were unable to find the root cause so we worked around it by adding a retry mechanism in our code (python) and whenever we hit a 403 we just retry...

It would be nice to know what might be causing this behaviour though

We're running Panorama 9.1 for reference