Issues with Overlay Routing and AWS Gateway Load Balancer

[aleksandar.astardzhiev]

Hey Folks,

I am having difficulties to get Overlay routing working with AWS GWLB and I was wondering is it something that I am doing wrong or missing some configuration element...

Any of you using AWS GWLB with overlay routing enabled?

In my test setup when overlay routing is enabled the test VM is able to reach internet over the PAN FW - Outbound is working fine.

But East-West between VPCs and Inbound traffic is not working. I can see traffic hitting the firewall, but allow traffic log show only byte send and no return traffic. Packet capture on the destination (for both east-west and inbound) doesn't show traffic to be arriving, so it looks like once FW inspect the packet and send back to GWLBe it doesn't send it in the correct direction.

If overlay routing is disabled everything works - east-west, inbound and outbound.

I found some old discussions mentioning issues with overlay routing, but from what I understand those know issues were for version 10.0.x, while we have tested with 10.2.1 and 10.1.6

NGFW AWS 

[mohamedridha]

I don't see it released?

[Centogene]

@npandey When is it released into the 10.2.x streams? 10.1.7 is out-of-date...

[JHall15]

I still don't see any indication 10.1.7 has been released let alone anything new in 10.2.x.  

[Mandanajan]

Just another thing to check, make sure you have enabled the Appliance Mode feature on the Transit Gateway, this feature ensures symmetric bidirectional traffic forwarding between VPC attachments. In other words, the forward and reverse flows are sent to the same firewall instance the same AZ for the lifetime of that flow. This allows firewalls to see both directions of the given flow thereby maintaining stateful traffic inspection capability inside Appliance VPC.

If the EC2 instances are deployed in two different VPCs in two different AZs. When the instances try to communicate with each other through AWS Transit Gateway with VPC attachments that are not in the same AZs results in asymmetric routing of packets. Forward flow and reverse flows from the same two communicating nodes, go to two different firewall instances in two different AZs, and the traffic is disrupted. This happens because, by default, when traffic is routed between VPC attachments, AWS Transit Gateway keeps the traffic in the same AZ where it originated until it reaches its destination.

[Mandanajan]

Appliance Mode is disabled by default on the VPC attachments in AWS Transit Gateway. For VPC-to-VPC traffic inspection through a Security VPC, you are required to enable Appliance Mode on the VPC attachment connected to the Security VPC. However, enabling Appliance Mode is optional for inspection of traffic originating from a spoke VPC destined to the Internet via dedicated Egress VPC.

[Centogene]

@Mandanajan 

The appliance modes are enabled, of course...they have to be! But more to the point: how otherwise would downgrading to 10.1.5-h5 work?! How would outbound flows work? but not East-West traffic flows?

In summary: it is a firmware version issue - not an appliance issue or configuration issue.

BW

[alan-griffiths]

Just hit this issue while running 10.1.6. It's reportedly fixed in 10.1.6-h6 and I can confirm this having upgraded.

https://docs.paloaltonetworks.com/pan-os/10-1/pan-os-release-notes/pan-os-10-1-6-known-and-addressed... 

Didn't stop me wasting a whole morning debugging it though...

[aleksandar.astardzhiev]

Hey Folks,

I haven't tracked this topic for very long time, it feels good to know that you are not insane and other are having the same problems...

Thank you @npandey for the insight. I can see 10.1.7 are officially released today and it looks like it is fixing the problem with the overlay routing:

https://live.paloaltonetworks.com/t5/vm-series-in-the-public-cloud/issues-with-overlay-routing-and-a...

PAN-194776

Fixed an issue on Amazon Web Services (AWS) Gateway Load Balancer (GWLB) deployments with overlay routing enabled where intra-zone packets were re-encapsulated with the incorrect source/destination MAC address.

I probably will not be able to test it anytime soon, because we already agreed on the design with disabled overlay routing and wouldn't change it soon.

[hfsantos]

Hey Folks,

Can anyone tell if you have already tested this version 10.1.7 with the fix?

[AsiaPac-Peter]

In my testing Only PANOS-10.1.7 not issue.

I test PAN-11.0.0 and 10.2.2 and 10.2.3-h2.

all have issue.

[OSTConsultancy]

Hello Palo community,

Has this been verified beyond 10.1.7 yet? obviously the version is outdated, and need Palo to look into this and confirm. @npandey  ?

[daveell]

Did you ever find a solution?

[OSTConsultancy]

Given the root cert issues expiring 31st December 2023 https://live.paloaltonetworks.com/t5/customer-advisories/emergency-update-required-pan-os-root-and-d...

We are forced to suck and see. I will update here - once we know what versions works. Or if we have issues.

Best wishes