topic Re: PA3050 cant ping next hop and has dropped all client traffic heading outbound. in General Topics

PA3050 cant ping next hop and has dropped all client traffic heading outbound.

lschs-s — Wed, 03 Apr 2019 02:57:09 GMT

I have tried a lot, and at this point I think I just must be missing something obvious that for whatever reason wont come to mind. From the PA3050 I can not ping outbound from the public IP. When I run captures, all outbound traffic is in dropped stage. There is no network functionality at all, and I am unable to find the issue.

Security ConfigNAT Config

Re: PA3050 cant ping next hop and has dropped all client traffic heading outbound.

Raido_Rattameister — Wed, 03 Apr 2019 14:24:29 GMT

Assuming that your public IP is 1.2.3.4

If you want to ping Google DNS then command would be:

> ping source 1.2.3.4 host 8.8.8.8

As you don't have External zone to External zone rule this traffic will match intrazone-default policy.

By default those policies don't log.

Click on intrazone-default and then override at the bottom.

Open intrazone-default policy and check "Log at Session End" on Actions tab to gain visibility.

Do the same with interzone-default.

Do you now see blocked sessions in Traffic log?

Re: PA3050 cant ping next hop and has dropped all client traffic heading outbound.

lschs-s — Wed, 03 Apr 2019 18:22:23 GMT

I am aware of how to ping, but its just not working. I enabled logging oneven more of the security policies and saw what I have seen before in the traffic tab. Connections seem to never complete and they always age out and application is left incomplete. I am unsure of where to go from here as this issue has left me quite confused. Whether it is an issue with NAT, Security, or some other rule, I need some help sorting this out.

Edit: Removed network's public IP and replaced it with a red square.

Re: PA3050 cant ping next hop and has dropped all client traffic heading outbound.

Raido_Rattameister — Wed, 03 Apr 2019 18:20:05 GMT

Last screenshot shows only sessions from internet towards your public IP.

No log of you initiating sessions from inside to internet or ping from firewall public ip to internet.

Also none of those incoming sessions match to your NAT policies.

Re: PA3050 cant ping next hop and has dropped all client traffic heading outbound.

lschs-s — Wed, 03 Apr 2019 18:26:52 GMT

They dont match mine, but they match intrazone which should let them through, right? Or due to intrazone being intra would it block it from finishing session with the external host? Also sorry, but the results are the same internally as well.

Re: PA3050 cant ping next hop and has dropped all client traffic heading outbound.

Raido_Rattameister — Wed, 03 Apr 2019 18:34:09 GMT

Can you add "packets sent" and "packets received" columns to the view?

Ping and DNS can be identified from first packet that is sent to client to server so from screenshot it is unclear if you receive any traffic back.

Also add "NAT Source IP" and verify if SNAT is applied to outgoing traffic and that you see your public IP in this column.

Re: PA3050 cant ping next hop and has dropped all client traffic heading outbound.

lschs-s — Wed, 03 Apr 2019 19:34:29 GMT

Src Internal

Src External

Re: PA3050 cant ping next hop and has dropped all client traffic heading outbound.

Raido_Rattameister — Wed, 03 Apr 2019 20:14:25 GMT

Weird. SNAT is applied.

Edit: Do you have correct Next Hop IP in virtual router? Can you ping next hop from fw external IP?

Can you click on magnifying glass or add egress interface column to verify that traffic is sent towards correct interface?

Under Network > Zones check how if only one outside interface is in External-Zone zone.

In cli add filter:

> debug dataplane packet-diag set filter off
> debug dataplane packet-diag clear filter all

> debug dataplane packet-diag set filter match source 8.8.8.8

> debug dataplane packet-diag set filter match destination 8.8.8.8

Run following command few times and check if severity is drop anywhere.

It will show what happened to traffic to and from 8.8.8.8 between periods you ran the command (filter delta yes)

> show counter global filter delta yes packet-filter yes

Clean up filter

> debug dataplane packet-diag set filter off
> debug dataplane packet-diag clear filter all

Last step would be to go with flow basic

https://knowledgebase.paloaltonetworks.com/KCSArticleDetail?id=kA10g000000ClS9CAK

Re: PA3050 cant ping next hop and has dropped all client traffic heading outbound.

lschs-s — Wed, 03 Apr 2019 20:34:37 GMT

Ill hold off on the CLI commands just so I can confirm with you, next hop is not reachable from the palo alto even when running "ping source <pub ip on extern zone> host <next hop ip>"

Re: PA3050 cant ping next hop and has dropped all client traffic heading outbound.

Raido_Rattameister — Wed, 03 Apr 2019 20:37:10 GMT

> ping source <external ip> host <next hop>

> show arp ethernet1/1 (assuming 1/1 is your external interface).

Next hop might have ping disabled but IP to mac resolution should still work.

If mac is not there then ask what is correct next hop from your ISP.

Re: PA3050 cant ping next hop and has dropped all client traffic heading outbound.

lschs-s — Wed, 03 Apr 2019 20:40:02 GMT

Also, yes, egress is correct.

Re: PA3050 cant ping next hop and has dropped all client traffic heading outbound.

lschs-s — Wed, 03 Apr 2019 20:44:17 GMT

Ping is not disabled, and ARP is incomplete on resolving. I recall seeing in either recieve or drop packet captures, ARP packets defining what the next hop's MAC was. Also another thing to note is my interface is a eth channel, but this issue was happening before as well.

Re: PA3050 cant ping next hop and has dropped all client traffic heading outbound.

Raido_Rattameister — Wed, 03 Apr 2019 20:50:51 GMT

If you disconnect Palo, connect cable to laptop, configure same IP and default gw IP to laptop can you ping next hop and get connectivity to internet?

Re: PA3050 cant ping next hop and has dropped all client traffic heading outbound.

Raido_Rattameister — Wed, 03 Apr 2019 20:53:37 GMT

So you have more than 1 cable to ISP and you have configured aggregated interface?

Re: PA3050 cant ping next hop and has dropped all client traffic heading outbound.

lschs-s — Wed, 03 Apr 2019 21:49:06 GMT

In regards to the dual link, in simplest terms, yes. However, in reality we own a /28 range and have a central router for just this range. From that router we have 2 links to a firewall plugged in for production networks, and another 2 links to the palo alto. Both of these firewalls have different public IPs.

As for the laptop, I will try that tommorow as currently I am out of the office. I have remote access to test other ideas, but as for physcial changes, I can not work on those now.

Re: PA3050 cant ping next hop and has dropped all client traffic heading outbound.

Raido_Rattameister — Wed, 03 Apr 2019 22:24:59 GMT

Have you enabled LACP on ae.x interface in Palo?

Re: PA3050 cant ping next hop and has dropped all client traffic heading outbound.

lschs-s — Wed, 03 Apr 2019 23:07:51 GMT

Yes and on both routers both links are active and LACP established the bond. Like I said, the issue was happening before the AE, so I doubt it has any influence on my issues here.

Re: PA3050 cant ping next hop and has dropped all client traffic heading outbound.

lschs-s — Thu, 04 Apr 2019 15:14:20 GMT

Did the test as you recomended, disabled LACP and did a direct L3 on the Palo Alto to my laptop. No client packets or pings from source 67.107.166.142 made it outbound. Its like the firewall isnt routing or has some setting blocking it from talking.

Re: PA3050 cant ping next hop and has dropped all client traffic heading outbound.

Raido_Rattameister — Thu, 04 Apr 2019 16:06:09 GMT

debug dataplane packet-diag set filter off
debug dataplane packet-diag clear filter all

debug dataplane packet-diag set filter match source 67.107.166.142
debug dataplane packet-diag set filter match destination 67.107.166.142

debug dataplane packet-diag set filter on

show counter global filter delta yes packet-filter yes

ping source 67.107.166.142 host <ip of your laptop>

show counter global filter delta yes packet-filter yes

Now post here output of the last show counter global result.

And then to clean up:

debug dataplane packet-diag set filter off
debug dataplane packet-diag clear filter all

Re: PA3050 cant ping next hop and has dropped all client traffic heading outbound.

lschs-s — Thu, 04 Apr 2019 18:29:01 GMT

Thank you so much for the help, but I fixed it all! It was some issues with subnetting and a few with routing, but worked them all out. I would go in detail, but it was in no way related to nat or security.