Not able to ping ISP B interface -10.2.9-h1

[UtkarshKumar]

Hi team,

We are facing an issue where we are not able to ping the secondary ISP's external interface when the default route is set for the primary ISP to take preference.

We have two ISPs: ISP A and ISP B.

The metric is set to prefer the ISP A route. When we try and ping the external interface of ISP B, we can see a weird behavior where when traffic comes on ISP B's external interface - in packet captures we can see the firewall does an echo reply from the same ISP B's interface but it changes in transmit stage of pcaps.

In traffic captures we can see that when we are in receive, Firewall stage the Source MAC is of ISP B's interface-- as expected but as soon as it comes to the transmit stage of the firewall the source MAC changes to ISP A's interface mac address. 

We have verified there is no PBF, we have the IP under permit IP addresses, and also have the right MGMT profile attached to the interface.

No NAT rule to perform a NAT on traffic originating from outside. We checked under routes we can see there is only 1 Static default route pointing toward ISP A. 

If we do a failover the same scenario will get replicated for the primary ISP link where we would not be able to ping the ISP A external interface. 

Question is why firewall would change the Source mac address/interface after coming to TRANSMIT stage. The PANOS is 10.2.9-h1

Can someone help in identify if we are missing anything or if there is something we can check?

[UtkarshKumar]

Hi team, did anyone faced a similar issue?

[SutareMayur]

Hi @UtkarshKumar Looking at the issue description, it looks it is happening because symmetric return is not happening.

Can you confirm how your two ISPs are load balanced ? Are you using ECMP ?

If you have ECMP enabled, you need to make sure symmetric return is enabled.

If you are not using ECMP, refer this article. This is for kind of a same scenario but for destination NAT. You can follow steps as per your use case.

Let us know how it goes!

[aleksandar.astardzhiev · Accepted solution]

Hi @UtkarshKumar ,

This is expected behaviour. Although this is considered "local" traffic (targeting the firewall itself, not passing through). Firewall will still perform policy and route lookup when generating reply for traffic to Data Plane interface.

Why this is happening:

1. As described in Getting Started: Packet Capture - Knowledge Base - Palo Alto Networks and Building Blocks for a Custom Packet Capture (paloaltonetworks.com) the four stages for the packet captures are:
   1. drop

      - When packet processing encounters an error and the packet is dropped.
   2. receive - When the packet is received on the dataplane processor.
   3. transmit - When the packet is transmitted on the dataplane processor
   4. firewall

      - When the packet has a session match or a first packet with a session is successfully create
2. These stages can roughtly map as follow to the flow sequence - Packet Flow Sequence in PAN-OS - Knowledge Base - Palo Alto Networks
   1. receive = ingress stage
   2. firewall = firewall session lookup + session setup + AppID + Content inspection
   3. trasmit = forward/eggress stage
3. As you can see during firewall stage, firewall perform route lookup to identify the destination/egress interface.
4. Since the default route for ISP-A is with better metric, only that route is installed in the FIB (forwarding table).
5. And since the FIB lookup return only ISP-A default route, the reply packet is forwarded through the interface connected to ISP-A

Suggested Solutions:

1. The simplest solution would be to create static host route for the source IP via ISP-B. This way reply will take the host route (since it is better match then the default route). The disadvantage of this approach is that ping to ISP-A from the same source IP will now fail, because replies will always go via ISP-B
2. More scalable solutions would require a bit more configuration. Note: I haven't tested, this but I am almost certain that it will work that way
   1. Enable ECMP and create two default routes via ISP-A and -B using same metric. This way both default routes will be installed in the FIB
   2. Enable "Symmetric Return" for the ECMP. This will ensure firewall will reply with the same interface, which has received the original packet
   3. Enabling ECMP however will cause your outbound traffic to be load-balanced between the two ISP, which you don't want. To avoid this you will need PBF (policy based routes) with path monitor, which will make sure that outbound traffic is forwarded via ISP-A and when path monitor is  down, traffic will be failovered to ISP-B

[UtkarshKumar]

Thanks @aleksandar.astardzhiev . I too thought the same but then we see:

https://knowledgebase.paloaltonetworks.com/KCSArticleDetail?id=kA10g000000ClF5CAK

Where it's mentioned This feature forwards the packet to the MAC address from where the SYN or lost packet was received. This ensures return traffic follows the same interface which the session created and is useful in asymmetric routing or Dual ISP environments.

Is it not correct that based on this we should see the same interface reply to pings and not follow the routing table?

[aleksandar.astardzhiev]

Hi @UtkarshKumar ,

As I mentioned in the KB and also in my post, this feature is enabled only when ECMP is enabled. From your original post it looks like you have two default routes with different metric. Even if ECMP is enabled for the virtual-router (VR), only one of the routes will be installed in the FIB if they have different metrics.

If you have ECMP enabled and configure both routes with the same metric, than this feature will allow you to receive reply from the same interface from which the request is received.

[glennne]

I am also encountering this issue. We're experiencing the same problem where the firewall changes the Source MAC address/interface after reaching the TRANSMIT stage when pinging the secondary ISP's external interface. Moreover, if you want to download capcut then click this given link capcutproapk.org. We've verified the settings, including the default route and lack of PBF, and we're facing the same behavior. Any insights or solutions would be greatly appreciated.

[aleksandar.astardzhiev]

Hi @glennne ,

What do you mean by "verfied the settings"?

In my previous post I have tried to explain why this is expected behaviour if you have two default routes with different metric and/or ECMP is not enabled.

Can you share little bit more of your configuration?