06-06-2022 06:13 AM - edited 06-06-2022 06:15 AM
I am having difficulties to get Overlay routing working with AWS GWLB and I was wondering is it something that I am doing wrong or missing some configuration element...
Any of you using AWS GWLB with overlay routing enabled?
In my test setup when overlay routing is enabled the test VM is able to reach internet over the PAN FW - Outbound is working fine.
But East-West between VPCs and Inbound traffic is not working. I can see traffic hitting the firewall, but allow traffic log show only byte send and no return traffic. Packet capture on the destination (for both east-west and inbound) doesn't show traffic to be arriving, so it looks like once FW inspect the packet and send back to GWLBe it doesn't send it in the correct direction.
If overlay routing is disabled everything works - east-west, inbound and outbound.
I found some old discussions mentioning issues with overlay routing, but from what I understand those know issues were for version 10.0.x, while we have tested with 10.2.1 and 10.1.6
07-27-2022 12:54 AM
07-28-2022 05:44 AM
I still don't see any indication 10.1.7 has been released let alone anything new in 10.2.x.
08-05-2022 01:24 PM
Just another thing to check, make sure you have enabled the Appliance Mode feature on the Transit Gateway, this feature ensures symmetric bidirectional traffic forwarding between VPC attachments. In other words, the forward and reverse flows are sent to the same firewall instance the same AZ for the lifetime of that flow. This allows firewalls to see both directions of the given flow thereby maintaining stateful traffic inspection capability inside Appliance VPC.
If the EC2 instances are deployed in two different VPCs in two different AZs. When the instances try to communicate with each other through AWS Transit Gateway with VPC attachments that are not in the same AZs results in asymmetric routing of packets. Forward flow and reverse flows from the same two communicating nodes, go to two different firewall instances in two different AZs, and the traffic is disrupted. This happens because, by default, when traffic is routed between VPC attachments, AWS Transit Gateway keeps the traffic in the same AZ where it originated until it reaches its destination.
08-05-2022 01:31 PM
Appliance Mode is disabled by default on the VPC attachments in AWS Transit Gateway. For VPC-to-VPC traffic inspection through a Security VPC, you are required to enable Appliance Mode on the VPC attachment connected to the Security VPC. However, enabling Appliance Mode is optional for inspection of traffic originating from a spoke VPC destined to the Internet via dedicated Egress VPC.
08-08-2022 02:47 AM
The appliance modes are enabled, of course...they have to be! But more to the point: how otherwise would downgrading to 10.1.5-h5 work?! How would outbound flows work? but not East-West traffic flows?
In summary: it is a firmware version issue - not an appliance issue or configuration issue.
09-06-2022 07:42 AM
Just hit this issue while running 10.1.6. It's reportedly fixed in 10.1.6-h6 and I can confirm this having upgraded.
Didn't stop me wasting a whole morning debugging it though...
09-14-2022 01:00 AM
I haven't tracked this topic for very long time, it feels good to know that you are not insane and other are having the same problems...
Thank you @npandey for the insight. I can see 10.1.7 are officially released today and it looks like it is fixing the problem with the overlay routing:
Fixed an issue on Amazon Web Services (AWS) Gateway Load Balancer (GWLB) deployments with overlay routing enabled where intra-zone packets were re-encapsulated with the incorrect source/destination MAC address.
I probably will not be able to test it anytime soon, because we already agreed on the design with disabled overlay routing and wouldn't change it soon.
02-27-2023 10:56 PM
In my testing Only PANOS-10.1.7 not issue.
I test PAN-11.0.0 and 10.2.2 and 10.2.3-h2.
all have issue.
Click Accept as Solution to acknowledge that the answer to your question has been provided.
The button appears next to the replies on topics you’ve started. The member who gave the solution and all future visitors to this topic will appreciate it!
These simple actions take just seconds of your time, but go a long way in showing appreciation for community members and the LIVEcommunity as a whole!
The LIVEcommunity thanks you for your participation!