PA Firewall VM series | interface suddenly goes down

[Fariq_Zaidi]

Hi Team,

We have a PA VM firewall where all working well but firewall interface (eth1/1 and eth1/2) went down suddenly

No issue with the management interface so we are still able to access the firewall during the incident time. 

to brings the interface up, we do reboot on firewall and thus fix the problem.

From the system log, we can see "link change" but customers stated that no healthy event was found from Azure.  (see attachment) 

Firmware: 10.1.9-h1  - check for known issue but not sure related with PAN-115816
(Microsoft Azure only) There is an intermittent issue where an Ethernet (eth1) interface does not come up when you first boot up the firewall.
Workaround: Reboot the firewall.

here the logs information found

brdagent.log
2023-07-11 04:50:29
2023-07-11 04:50:29.097 +0300 Port 1: Down 10Gb/s-full duplex
brdagent.log
2023-07-11 04:50:29
2023-07-11 04:50:29.099 +0300 PORT1: board_port_autoneg_enabled -> board_port_autoneg, link: 0, mode: 1
brdagent.log
2023-07-11 04:50:29
2023-07-11 04:50:29.102 +0300 port status changed/updated
brdagent.log
2023-07-11 04:50:29
Port 1: mode changed to autoneg
brdagent.log
2023-07-11 04:50:29
2023-07-11 04:50:29.106 +0300 port status changed/updated
brdagent.log
2023-07-11 04:50:29
Port 1: mode changed to autoneg
brdagent.log
2023-07-11 04:50:29
2023-07-11 04:50:29.109 +0300 Error: vm_is_azure_gwlb(vm/vm_ports.c:817): sysd_fetch_obj() failed for cfg.net.azr-gwlb
brdagent.log
2023-07-11 04:50:29
2023-07-11 04:50:29.444 +0300 Port 2: Down 10Gb/s-full duplex
brdagent.log
2023-07-11 04:50:29
2023-07-11 04:50:29.447 +0300 PORT2: board_port_autoneg_enabled -> board_port_autoneg, link: 0, mode: 1
brdagent.log
2023-07-11 04:50:29
2023-07-11 04:50:29.447 +0300 port status changed/updated
brdagent.log
2023-07-11 04:50:29
Port 2: mode changed to autoneg
brdagent.log
2023-07-11 04:50:29
2023-07-11 04:50:29.455 +0300 port status changed/updated
brdagent.log
2023-07-11 04:50:29
Port 2: mode changed to autoneg
brdagent.log
2023-07-11 04:50:29
2023-07-11 04:50:29.456 +0300 Error: vm_is_azure_gwlb(vm/vm_ports.c:817): sysd_fetch_obj() failed for cfg.net.azr-gwlb
brdagent.log
2023-07-11 04:50:31
2023-07-11 04:50:31.459 +0300 Error: vm_is_azure_gwlb(vm/vm_ports.c:817): sysd_fetch_obj() failed for cfg.net.azr-gwlb
brdagent.log
2023-07-11 04:50:31
2023-07-11 04:50:31.794 +0300 Error: vm_is_azure_gwlb(vm/vm_ports.c:817): sysd_fetch_obj() failed for cfg.net.azr-gwlb

Please let me know if we can find out the Root cause why interface down and connection cannot resume until we need to do reboot to get this fix.

[BPry]

@Fariq_Zaidi,

If you're looking for root cause I'd recommend that you engage TAC so that they can get a technical support dump and actually see all of the associated logs. Given what you have visible in the logs you've shared the firewall detected the interface went down, however that's just based off of a single file entry.

I'll just toss out there though that troubleshooting a NIC issue on Azure is absolutely horrific, so I'm not sure how your customer could really argue that the interface didn't or did drop momentarily. You simply don't have the logs on an Azure VM at that level to actually say the NIC didn't have an issue. 

[Fariq_Zaidi]

Update from Palo alto Team,

this issue i am facing is related with internal Bug ID and the fix will be in  PAN-OS 10.1.11 is expected to be released in the month of September. And no workaround is available. 

hope this help if anyone facing the same issue.

thank you

[Liviu_Bodeanu]

Hi @Fariq_Zaidi ,

I experienced the same type of issue in PAN-OS 10.1.6.
May you please let me know what is the bug ID under which PAN TAC identified this issue?

Many thanks,
Liviu

[Fariq_Zaidi · Accepted solution]

Hi Liviu,

PAN-219643 : https://docs.paloaltonetworks.com/pan-os/10-1/pan-os-release-notes/pan-os-10-1-11-known-and-addresse...

Regards

Fariq