Automating the Palo Alto NGFW's Process/Deamon Restarts

[nikoolayy1]

The Palo Alto NGFW has a great API interface and there is even an integrated tool to view the API commands, called api browser that is located at the <firewall ip>/api and it is described at Use the API Browser (there is even a debug window for API traffic. You can also read Use the Web Interface to Find XML API Syntax or Use the CLI to Find XML API Syntax ) but it does not provide access to the operational commands that are starting with "debug xxxx" but why do we need access to those commands?

The answer to the previous question is to restart some palo alto processes without rebooting the full firewall.

For example "debug software restart process web-server" is to restart the backend web-server that is responsible for the PAN-OS GUI. I also suggest checking the articles below:

• Knowledge sharing: restarting palo alto processes, reboot, shutdown, factory default reset (authored by me)
• Commonly Used Processes/Daemons
• How to Restart the Management server "mgmtsrvr" Process
• How to Restart the Web-related Processes

The issue that the API does not have access to the debug commands is that we can't use something like Ansible "paloaltonetworks.panos.panos_op module" which allows to execute random operational commands with Ansible. Ansible has a separate module to run random config commands called panos_config_element and another one called panos_type_cmd (panos_type_cmd seems older one and does not do good checks when pushing commands). We also can't use the Palo Alto XSOAR PAN-OS integration that is shown here, but the XSOAR RemoteAccessV2 is another option to push a debug commands, using SSH, to reboot a process.

As a note, Terraform is more limited to triggering custom commands with its Resources (similar to Ansible modules with tasks) on the Palo Alto Firewalls compared to Ansible (panos_type_cmd/ panos_config_element and panos_op) or Cortex XSOAR (the PAN-OS or RemoteAccessV2 integrations) as Terraform's idea is to have predefined Resources and there is Terraform resource like in Ansible to send custom Operational or Type (configuration) commands if they are available through the Palo Alto API, so only the predefined resources can be used and you can check out this page from Terraform.

Example Playbook for operational commands (except debug commands). For everything else Ansible and XSOAR are still good options (you may need to do things like"pip install pan-os-python " as for some reason when the Ansible collection is installed this is missing.

---

- name: Palo Alto Provision
  hosts: palo
  connection: local
  gather_facts: false
 
  vars:
provider:
   api_key: xxxxxx
   ip_address: "{{ ansible_host }}"
 
  collections:
- paloaltonetworks.panos
 
  tasks:
- name: show list of all interfaces
   panos_op:
     provider: '{{ provider }}'
     cmd: 'show interface all'

The solution to this issue are the good old TCL Expect scripts! Some may not know but this is one of the first ways to do automation by using SSH even before API was an option. As noted leave the "management-server" process for last if you need to restart it together with other processes as it can kill the script's ssh connection!

Example of a TCL expect script (when the management server is restarted nothing is seen in the CLI, so we can't use "expect" and the ssh connection is terminated, so this process should be restarted at the end):

#!/usr/bin/expect -f

# Get the commands to run, one per line
set timeout 60
spawn $env(SHELL)
set DEBUG 1
set USER xxxx
set PASS xxxx
set IP_AD xxxx

spawn ssh $USER@$IP_AD
match_max 100000
expect "*?assword:*"
send -- "$PASS\r"
sleep 2
expect "*>*"
send -- "set cli terminal width 200\r"
sleep 2
expect "*>*"
send -- "set cli scripting-mode on\r"
sleep 2
expect "*>*"
send -- "set cli terminal type xterm\r"
sleep 2
expect "*>*"
send -- "debug software restart process web-backend\r"
expect "*was restarted*"
sleep 2
expect "*>*"
send -- "debug software restart process web-server\r"
expect "*was restarted*"
sleep 2
expect "*>*"
send -- "debug software restart process sslvpn-web-server\r"
expect "*was restarted*"
sleep 2
expect "*>*"
send -- "debug software restart process management-server\r"
expect eof

"expect eof" is only needed for the management-server restart at the end of the scrip as it not always returns "was restarted" and this will keep the connection open until the management server restart kills it or the script times out.

The TCL Expect script can be on a linux server and Ansible and Cortex XSOAR when there is an Incident and logs in the Cortex Data Lake with its RemoteAccessv2 integration can trigger it as this seems a great integration!