Major issue PanOS 8.1.3: Network intefaces go down

[ZEBIT]

We have some issues for some users with the globalprotect vpn to connect to our PA-3260 firewall.
To solve this issue technical support told us to upgrade to our PanOS from 8.1.2 to 8.1.3.

We did this morning and everything went fine till 1PM.

From some users we received there was a connectivity issue to internet, other users told us they couldn't connect to servers public IP behind the firewall.

Start troubleshooting the whole network we could see in our arp table of the router that mac-addresses of the firewall had an "incomplete" ip address.

Looking further in this issue we found that the firewall didn't send any ARP messages anymore to any device and due that the external servers couldn't be reached through the router.

We also found that GUI was reacting very slow.

After 4 hours of troubleshooting with the technical support from the Palo Alto partner we decided to go back to version 8.1.2.

We restarted the firewall and all the issues above were solved.

I think Palo Alto need to have a look into the 8.1.3 release.

[bspilde]

I've had repeated issues with ARP on public interfaces of the PA firewall. In my specific case we have two separate interfaces one for guest and one for corporate public internet access. The firewall randomly decides to send ARP replies from the wrong interface. Therefore we have made it a standard process that any and all non-guest interface IP's including new NAT entries have to be static ARP assignments on the public routers. This was supposedly fixed in a release long ago but we recently determined it still exists.

@SteveMcCall

[BPry]

@ZEBIT,

Anytime TAC tells you to upgrade to the latest release the first things you should be asking them is the following:

1) Is this a recommended release (As of this moment I don't believe 8.1.3 is recommend). 

2) What is the Issue ID you believe I fall under? 

3) If its still an internal bug ID, please share that with me under an NDA if necissary. 

The only GlobalProtect issues  addressed in 8.1.3 is PAN-96326 (OCSP status issue when using certificate authentication), PAN-93864 (password field not displaying when certificate profile is attached to the portal configuration), PAN-91926 (Premature deletion of proxy sessions), PAN-90535 (unnecessarily sent an Authroize-only request to Radius server). It's possible that you may have been hitting PAN-96326 or PAN-90535; 90535 you can work around fairly easily. 

If TAC can't share the issue ID they believe that you fall under or an internal bug report that has yet to be verified I generally would not recommend following any willy nilly upgrade your os suggestion. Almost every vendor TAC has an instance of using this when they essentially reach that '**bleep** I don't know' level, and it's not helpful in the troubleshooting process at all. There's exceptions to this of course, like if you were running 7.0.3 or some other severly outdated release, but you were running the current recommended release as you should have been. 

That being said, 8.1.3 has been increadibly stable on my fleet of test devices and smaller remote offices so far and I don't believe that this is an issue that has been reported to the community as of yet. Regardless, thanks for making a post so that others can keep an eye out for it. 

[bspilde]

I would recommend doing a static ARP entry for your public server IP addresses on the public routers.

[Brandon_Wertz]

I would agree with you @BPry except for in this case that the 3200 series firewalls will only run 8.1.X code.  I would agree you should perform your best to look try to not upgrade out of one bug and into another, but in this case I think it might be unavoidable.

This is probably a scenario where leveraging your SE and pushing for "behind the screens knowledge" is the preferred route.

[vineethp101]

I changed ASA with Palo alto.I used PanOS 8.1.3.Palo Alto DMZ zone and WAN Zone is connected to WAN Switch. All devices connected to WAN switch is working fine. But palo alto connot learn ARP through this switch and shows incomplete mac entry. But when i connect ISP router and one DMZ server directctly to palo alto then learned mac address. What may be the issue

[AnalysisMan]

The TAC didn't recommend PAN-OS 8.1.2 and 8.1.3. Your issue might be related to the following defect. I will go with 8.1.1 this weekend.

Hope this helps!

#

Here is the defect PAN-101604. Please verify if it is applicable in your scenario OR not.

[8.1.2 PA-3220] Firewall sends ARP broadcast request with wrong Sender/Target IPs , with "pan_plfm_fe_cp_arp_delete(src/fe/cp/pan_platform_fe_cp.c:1544): Cannot find such an entry"

1) FW sends ARP request broadcast for its own interface, with neighbor's IP as sender's IP address, triggered by neighbor's OSPF update message.
2) "show arp all" shows entries for FW's own interfaces. These entries are marked as "incomplete."

Workarounds: Disable OSPF if possible OR downgrade to 8.1.1.

[vineethp101]

Thanks for reply. We are not using ospf. We used a layer 2 switch between isp and palo alto. Dmz machines are also in same switch. Before with asa everything was working fine. But palo alto can’t learn mac through this switch. If bypass this switch everything works fine. All are fast ethernet ports in this switch. I tried another fast ethernet switch same issue.

[Brandon_Wertz]

Unless you're running a 3200 series FW which require using PAN-OS 8.1.X people really shouldn't be running 8.1.X until at least patch 5.

[vineethp101]

Am using PA 3220

[AnalysisMan]

FYI - PAN-OS 8.1.4 is released today and available, but I don't see any addressed issues with ARP.

https://www.paloaltonetworks.com/documentation/81/pan-os/pan-os-release-notes/pan-os-8-1-addressed-i...

[vineethp101]

PanOS 8.1.4 released. Addressed bug related to this discussion.
In PA 3200 ports 2,3,4,6,8,10 worked only in 1 G in 8.1.3

[vineethp101]

PanOS 8.1.4 released. Fixed bug related to this discussion. IN 8.1.3 ports 2,3,4,6,7,8,10 functioning only in 1 G. My wan switch have only fast ethernet ports.