Palo Alto not loading certain valid sites, why?

[drewdown]

Having another issue with these things (ready to throw them out the window in all honesty). 

Seems as though a couple sites simply won't load when routed through a pair of HA-3020s.  Of course Palo Alto support does a packet capture and sees no drops so immediately not their fault.  But I know it is because I can route that specific traffic out another gateway on the same public network the PAs sit on and it works without issue.  Packet captures from the client side show alot of TCP re-transmissions when the traffic is routed through the PAs, when its routed through an ASA I see 0 re-transmits.  PA wants to point the finger at the ISP but I highly doubt our ISP is having an issue routing two completely different website/IPs back to a single IP on the same /26 network.  No SSL decryption.  

Nothing in the logging on the Palo Alto shows any block/drop or deny.  Has anyone seen anything like this before? 

[Brandon_Wertz]

Your problem statement is a bit confusing.  Can you clarify a bit more, adding in architecture information as well as the websites in question which aren't working?

[drewdown]

Yeah sorry, just frustated with PA so more of a vent post.  Hopefully this helps:

I have a pair HA active/passive 3020s and users behind them are unable to access a couple sites.  If I add a static route for those sites on my core (cat6k) and make the next hop an ASA (on the same public network as my 3020s) the sites load without issue. 

Lets say the ASA has an IP of 11.11.11.3/26 and the PAs have 11.11.11.4/26 and the only thing in front of them both is an L2 DMZ switch.  When routing the traffic through the PAs I see a lot of tcp retransmissions, when I route it through the ASA I don't see any of those.  I would understand if it was an ISP issue to my entire /26 but its clearly not.  Only an issue when traffic is routed through the PAs.  

One of the sites is cigna.com 170.48.10.90 and you can see part of the captures below, ASA on top and PA on the bottom.  

[Brandon_Wertz]

Thanks for the clarification.  

I might not fully get it yet so bear with me.  The 3020s would these be "perimeter" firewalls, where really the next hop beyond them is an ISP or is there another set of firewalls beyond the 3020s?

If the 3020s are perimeter devices, the PA IP address of 11.11.11.4 is that your public hide / NAT address?  If so you'll want to make sure that the "object" entry for that IP is a /32.  While the 11.11.11.4 might be a part of a /26 network in order to use the .4 as your NAT it needs to be a /32 object in the firewall.

[drewdown]

Yes perimeter FWs, nothing allowing or denying traffic in front of them, next hop is one of my ISPs.  And I thought I was doing many to 1 NAT using object 11.11.11.4/26.   You are saying I should change that object to 11.11.11.4/32? If I do that is it going to complain about my default route being in a /26 and the outside/untrust IP being a /32?  Right now my outside/untrust ae1 and my outbound NAT statement both use the 'outside interface' object 11.11.11.4/26. 

For outbound NAT I am doing:

Dynamic IP and Port
Interface Address
ae1
outside object (11.11.11.4/26)

Which appears to be right because my public IP shows a 11.11.11.4. 

[Brandon_Wertz]

In my org it's easier to do static routing on the firewall so that's what I'll describe for our deployment.

Default route egressing the firewall goes to a /24 network (a .1 on said network) which is owned by our border routers.  We're utilizing a specific IP (/32 object entry) which exists in that /24 on the egress/external side of the firewall for our NAT.  

This is how Palo requires the object to be created in order to have traffic utilize an IP for a service.

[drewdown]

Makes sense but are you saying the way I have it could be causing me a problem?  I ask because this isn't our only HA PA environment and they are all setup the same way but only this pair having problems.   

[Brandon_Wertz]

@drewdown wrote:

Makes sense but are you saying the way I have it could be causing me a problem?  I ask because this isn't our only HA PA environment and they are all setup the same way but only this pair having problems.   

---

Yes, 5ish years ago when I started working on Palo I deployed the firewall with the object as /24, because it was a apart of a /24 network.  I couldn't get any Internet browsing to work.  I opened a case with Palo TAC and was told about this /32 requirement.

That said I would contact TAC again and point them to this point and let them confirm for your environment.

[drewdown]

I will but don't think that is my issue.  Thanks though.  

[carlebert]

Just wanted to chip in and say I had the same issue with a 220 (not in a HA config) a while back at one of my sites.  We tried everything we could think of as far as ensuring nothing on the PA was interfering with it, eventually I tried adding a rule to force it through a secondary WAN connection with PBF and the site loaded just fine.  Unfortunately I don't have the logs from that incident anymore and when I went to see if this was still the case the site now works with both ISPs there.  So in my case it likely was an issue on the ISP side (Airespring who was using Sprint's network I believe).

[drewdown]

So more and more sites stopped loading and looking at the logs I could see 0 bytes received when I tried to browse any of them.  On a whim I changed the outside IP of the PAs (changed it to 11.11.11.15/26) and that immediately fixed the issue.   So something upstream, ARP related, route or duplicate IP was stopping return traffic from reach the PAs.  

Honestly the last thing I would think the problem would be but at this point its fixed.   

[Brandon_Wertz]

---

@drewdown wrote:

So more and more sites stopped loading and looking at the logs I could see 0 bytes received when I tried to browse any of them.  On a whim I changed the outside IP of the PAs (changed it to 11.11.11.15/26) and that immediately fixed the issue.   So something upstream, ARP related, route or duplicate IP was stopping return traffic from reach the PAs.  

Honestly the last thing I would think the problem would be but at this point its fixed.   

---

Take it for what you will, but you really should reach out to TAC and ask them about using a /26 object for your NAT vice the /32.

I've accidentally gotten my site on Cox's "blacklist" because I didn't use the correct mask (ie using /26 for an object on a GP setting).  What ended up happening is my 5020 was ARPing out for all the hosts on the network I specified instead of just using the single IP I intended for the interface.

While the change of IP might have masked the problem, I don't believe it solved it...Just trying to let you benefit from my past mistakes.

[drewdown]

I agree, changing the IP fixed the problem at hand but didn't shed any light on why it was happening.  Will ask tac about the outbound NAT configuration and see what they say.