PAN-OS 10.1.9 -- CAUTION

[Brandon_Wertz]

I would highly recommend waiting on deploying 10.1.9 in your enviornment, whether it be large enterprise or SOHO.

Recently my company had upgraded a 5250 pair from 10.1.8 to 10.1.9 as we had experienced 2 service impacting bugs.

After upgrading 10.1.9 we found that the FW was exhibiting odd behavior where traffic which had specific allow rules for it wasn't matching the defined allow rules and was instead being denied all together or was matching a different rule lower in the policy stack.

What made troubleshooting this further was the traffic which was being denied, that shouldn't have been, could only be seen in the session browser as being denied.  Even hours later the incorrectly denied traffic was nowhere in the GUI logs; and could only be found in the CLI by using the session ID from the temporary session browser view while the session existed there.

We had a log P1 TAC case that unfortunately found no cause for this behavior so we downgraded back to 10.1.8 and all services were restored.

Traffic types effected were:

IKE

IPSec

GRE

Non-decrypted HTTPS TLS1.2 (That we saw)

[JayGolf]

Hi @Brandon_Wertz ,

Thanks for sharing! This is great feedback that can be helpful for our users. 

[BPry]

10.1.9-h1 was just released to the public, however it doesn't look like the release notes have been published just yet. 

[OtakarKlier]

OK so 10.1.8 is pretty good?

[Raido_Rattameister]

To piggyback on the topic.

March 7 dynamic update updated threat id 32968 that has loads of false positives in SMTP traffic and no fix out yet...

[dpeeters]

We are having the same problem with IKE and IPsec. After downgrading to 10.1.8 everything is working again.

[as-mg]

Interesting, we're seeing issues after upgrading to PAN-OS 10.1.9 as well, coming from 10.1.8-h2.

Both internal and external traffic has started to act weird, causing reconnects to on-premise Skype for Business as well as issues with an external cloud call center solution. I was unable to find anything in the known issues list that would explain this, but I'll try to downgrade back to 10.1.8-h2 to see if it is resolved there.

[Brandon_Wertz · Accepted solution]

This case went on forever unfrotunately, but we ultimately found one of the causes of our issues.  10.1.9 changed the behavior of how the firewall responded to an ARP.

Prior to 10.1.9 the firewall would respond back out an egress (outgoing) interface different from the ingress interface.  Which meant that a NAT policy could have an interface the that traffic didn't originate from.  In 10.1.9 Palo changed this behavior where the FW would (only send a pack back egress that it ingressed) match a NAT policy where the incoming and outgoing interfaces were the same.

Palo by changing this behavior ultimately made our NAT policy no longer work.  We updated our NAT policy to match an "ANY" outgoing interface instead of a specific one and doing so solved our upgrade issue with traffic failing after upgraded:

PAN-195114 -- though in our instance we have a single VR not 2 as the bug-ID mentions.

IMO -- This change in behavior shouldn't have been a "patch" and should have come in a major PAN-OS upgrade, not a minor release.  But Palo saw this as a "bug" hence they released it as a patch/minor release update.

[Raido_Rattameister]

So assuming your WAN interface runs on ethernet1/1 your DNAT policies (traffic from Internet to inside) had random destination interfaces attached?

Something like this?
Source zone - WAN
Destination zone - WAN

Destination interface - ethernet1/3

[Brandon_Wertz]

Kind of, but not really.

The external side of our FW is all L2.  We have multiple L2/L3 segments on our WAN/INet beyond the FW.  There are 2 Border routers beyond the firewall own the L2/L3, which the firewall sees as the WAN.  The BRs are EIGRP neighbors with HSRP for the multiple L2/L3.

So in the NAT policy we just "picked" a "random" L2 network as the egress interface, because for us it really doesn't matter if it's the "right" or matching ingress interface because the BRs own all of them.

So the BRs have:

WAN-1 (WAN-Zone)

WAN-2 (WAN-Zone)

WAN-3 (WAN-Zone)

The FW participates in WAN-1, WAN-2, WAN-3 from a L2 perspective.  The ingress might have come in on WAN-1, but the NAT might have said egress = WAN-2, and it worked because the next hop was the BR which has/knows about all those L2 segments.  

But with palo "fixing the bug" the egress NAT interface needed to match the specific interface even though they are all a part of the same zone..