topic Re: Not able to ping ISP B interface -10.2.9-h1 in Next-Generation Firewall Discussions

Not able to ping ISP B interface -10.2.9-h1

UtkarshKumar — Mon, 19 Aug 2024 19:12:01 GMT

Hi team,

We are facing an issue where we are not able to ping the secondary ISP's external interface when the default route is set for the primary ISP to take preference.

We have two ISPs: ISP A and ISP B.

The metric is set to prefer the ISP A route. When we try and ping the external interface of ISP B, we can see a weird behavior where when traffic comes on ISP B's external interface - in packet captures we can see the firewall does an echo reply from the same ISP B's interface but it changes in transmit stage of pcaps.

In traffic captures we can see that when we are in receive, Firewall stage the Source MAC is of ISP B's interface-- as expected but as soon as it comes to the transmit stage of the firewall the source MAC changes to ISP A's interface mac address.

We have verified there is no PBF, we have the IP under permit IP addresses, and also have the right MGMT profile attached to the interface.

No NAT rule to perform a NAT on traffic originating from outside. We checked under routes we can see there is only 1 Static default route pointing toward ISP A.

If we do a failover the same scenario will get replicated for the primary ISP link where we would not be able to ping the ISP A external interface.

Question is why firewall would change the Source mac address/interface after coming to TRANSMIT stage. The PANOS is 10.2.9-h1

Can someone help in identify if we are missing anything or if there is something we can check?

Re: Not able to ping ISP B interface -10.2.9-h1

UtkarshKumar — Wed, 21 Aug 2024 07:28:17 GMT

Hi team, did anyone faced a similar issue?

Re: Not able to ping ISP B interface -10.2.9-h1

SutareMayur — Wed, 21 Aug 2024 08:05:35 GMT

Hi @UtkarshKumar Looking at the issue description, it looks it is happening because symmetric return is not happening.

Can you confirm how your two ISPs are load balanced ? Are you using ECMP ?

If you have ECMP enabled, you need to make sure symmetric return is enabled.

If you are not using ECMP, refer this article. This is for kind of a same scenario but for destination NAT. You can follow steps as per your use case.

Let us know how it goes!

Re: Not able to ping ISP B interface -10.2.9-h1

aleksandar.astardzhiev — Wed, 21 Aug 2024 08:19:12 GMT

Hi @UtkarshKumar ,

This is expected behaviour. Although this is considered "local" traffic (targeting the firewall itself, not passing through). Firewall will still perform policy and route lookup when generating reply for traffic to Data Plane interface.

Why this is happening:

As described in Getting Started: Packet Capture - Knowledge Base - Palo Alto Networks and Building Blocks for a Custom Packet Capture (paloaltonetworks.com) the four stages for the packet captures are:
1. drop
  - When packet processing encounters an error and the packet is dropped.
2. receive - When the packet is received on the dataplane processor.
3. transmit - When the packet is transmitted on the dataplane processor
4. firewall
  - When the packet has a session match or a first packet with a session is successfully create
These stages can roughtly map as follow to the flow sequence - Packet Flow Sequence in PAN-OS - Knowledge Base - Palo Alto Networks
1. receive = ingress stage
2. firewall = firewall session lookup + session setup + AppID + Content inspection
3. trasmit = forward/eggress stage
As you can see during firewall stage, firewall perform route lookup to identify the destination/egress interface.
Since the default route for ISP-A is with better metric, only that route is installed in the FIB (forwarding table).
And since the FIB lookup return only ISP-A default route, the reply packet is forwarded through the interface connected to ISP-A

Re: Not able to ping ISP B interface -10.2.9-h1

UtkarshKumar — Wed, 21 Aug 2024 14:23:28 GMT

Thanks @aleksandar.astardzhiev . I too thought the same but then we see:

https://knowledgebase.paloaltonetworks.com/KCSArticleDetail?id=kA10g000000ClF5CAK

Where it's mentioned This feature forwards the packet to the MAC address from where the SYN or lost packet was received. This ensures return traffic follows the same interface which the session created and is useful in asymmetric routing or Dual ISP environments.

Is it not correct that based on this we should see the same interface reply to pings and not follow the routing table?

Re: Not able to ping ISP B interface -10.2.9-h1

aleksandar.astardzhiev — Wed, 21 Aug 2024 14:48:53 GMT

Hi @UtkarshKumar ,

As I mentioned in the KB and also in my post, this feature is enabled only when ECMP is enabled. From your original post it looks like you have two default routes with different metric. Even if ECMP is enabled for the virtual-router (VR), only one of the routes will be installed in the FIB if they have different metrics.

If you have ECMP enabled and configure both routes with the same metric, than this feature will allow you to receive reply from the same interface from which the request is received.

Re: Not able to ping ISP B interface -10.2.9-h1

aleksandar.astardzhiev — Fri, 23 Aug 2024 15:17:25 GMT

Hi @glennne ,

What do you mean by "verfied the settings"?

In my previous post I have tried to explain why this is expected behaviour if you have two default routes with different metric and/or ECMP is not enabled.

Can you share little bit more of your configuration?