#PA-820 Not routing to Internet

[GarrettMichaelHayes]

I have an issue with a PA-820 that has me scratching my head. I'll try to keep the description short, but this one takes some background.

The History:

I had two physical sites in different parts of the country, one with a PA-820 the other with a pair of PA-3020s.  I had to move the PA-820 and all of the infrastructure behind it into the same physical site as the PA-3020's.  The meant changing ISPs and address ranges on the PA-820.

I had a spare range at the new site, but the ISP had it statically routed to the main public IP address of the PA-3020s, and could not remove the static routing in time for the move.

So I came up with a strange short term solution, which worked.  I left the PA-820 configured as it was.  On the PA-3020s I configured an interface with the former public gateway address used by the PA-820.  I connected the outside interface of the PA-820 to this inside interface on the PA-3020s. Then I created a set of security and NAT rules to pass traffic between the new public range that was routed to the PA-3020s and corresponding public addresses on the PA-820.

This worked very well - the PA-820 didn't know it had gone anywhere new, and the rest of the world thought its services were now at a completely new set of IPs.  The PA-3020s just played courier for those addresses.

The Problem:

Last week, at our request, the ISP finally dropped the static routing for that "passthrough" address range.  We scheduled a window to move the PA-820's outside connection to the switch that already shares the ISP drop between the two PA-3020s, and to reprogram the PA-820 and the PA-3020s.

We removed the relevant addresses from the PA-3020s.  We added them to the PA-820 and removed the "old" ISP addresses. We changed the default route appropriately, etc.  And .....  Nothing.  Traffic is not passing through to the ISP or beyond.

Things I examined and tested:

Laptop Test - A laptop connected to the same switch port used by the PA-820, and using the same IP and gateway address communicates to the Internet with no problem.

The switch - The switch sees the MAC addresses of the active PA-3020, the PA-820 and the ISP's device all on the correct ports, all on the correct VLAN.

ARP - The ARP tables on the PA-3020 and the PA-820 show the same ISP device, each associated with the appropriate gateway address for that PA's IP range.

Pings and Traceroutes from both the PA-820 and from a VM behind it.  No ping works even to the gateway address.

NAT - Verified that the default NAT rule is configured with the correct interface and IP

Interface IPs  - PA-820 - Verified that the correct reassigned IPs are specified on the interface ( 1/1 "untrust")

Interface IPs - PA-3020 - Verified that the reassigned IPs are no longer on the PA-3020

Traffic Log - The traffic log on the PA-820 shows the traffic from the test VM being passed by the correct security policy, correctly natted and passed out the correct port.

Restart - Did a complete restart on the PA-820, just to make sure there wasn't some session or table entry that was "stuck".\

Packet Trace - A packet trace on the PA-820 shows the packets from the test VM being received, but then being dropped AFTER they have been natted.

I ended up reverting the changes and leaving the PA-820 in its "chained" configuration.

So, the PA-20 is pretty definitely dropping the packets, but I'm at a loss to understand why.

Any and all suggestions of other things to look at or things I might have missed will be greatly appreciated.

[aleksandar.astardzhiev]

Hi, @GarrettMichaelHayes ,

From my experience if traffic is allowed by the policy, but still dropped by the firewall, most common reason would be msiconfigured routing or zone protection. But it will difficult to guess without inspecting the actual configuration.

I cannot stress this enough, but the best way to understand the dropped packets is to use global counters with packet filters - https://knowledgebase.paloaltonetworks.com/KCSArticleDetail?id=kA10g000000CloNCAS

- Using the GUI set a packet filter for destination (it is better to filter for destination, because filtering by source, will not match the packets after source NAT) and I would also suggest to filter by protocol "1" - to inspect only ICMP traffic while troubleshooting (and limit the noise)

- Run the pings from VM machine to destination on internet that you are filtering on

- Via CLI check the global counters with "> show counter global filter packet-filter yes delta yes" ('packet-filter' will show data only for the traffic that is matching your filter, while 'delta' will show you only data between the last and previous execution fo the comman - so you need to execute the command at least twice with couple of seconds in between).

As you can imagine this will require to configure everything back again like you want to do - haveing another maintenance, but this is the best way to understand why was causing the packet drop.

In addition I would suggest you to double/triple check the following configuration:

- Default route: make sure default is pointing to correct next-hop IP and interface

- Source NAT: make sure your NAT is matching both destination zone and destination interface (or just use any for destination interface). If you use FW interface for translated IP, edit the translated source section and select the new IP from the dropdown.

- Zone protection: If you have enabled zone protection on the trust and untrust zone, check if your routing table is correct and anti-spoofing is not messing round. You may want to try to disable/remove zone protection from the zones just for the tests.

[GarrettMichaelHayes]

Thank you.  That's helpful information.  I will indeed have to schedule another maintenance window, which will have to be in the wee small hours of a weekend.  Not sure if it can be this weekend or next.  I'll follow up with my findings.

GMH