topic Re: Issues with Overlay Routing and AWS Gateway Load Balancer in VM-Series in the Public Cloud

Issues with Overlay Routing and AWS Gateway Load Balancer

A_Astardzhiev — Mon, 06 Jun 2022 13:15:23 GMT

Hey Folks,

I am having difficulties to get Overlay routing working with AWS GWLB and I was wondering is it something that I am doing wrong or missing some configuration element...

Any of you using AWS GWLB with overlay routing enabled?

In my test setup when overlay routing is enabled the test VM is able to reach internet over the PAN FW - Outbound is working fine.

But East-West between VPCs and Inbound traffic is not working. I can see traffic hitting the firewall, but allow traffic log show only byte send and no return traffic. Packet capture on the destination (for both east-west and inbound) doesn't show traffic to be arriving, so it looks like once FW inspect the packet and send back to GWLBe it doesn't send it in the correct direction.

If overlay routing is disabled everything works - east-west, inbound and outbound.

I found some old discussions mentioning issues with overlay routing, but from what I understand those know issues were for version 10.0.x, while we have tested with 10.2.1 and 10.1.6

Re: Issues with Overlay Routing and AWS Gateway Load Balancer

Mandanajan — Wed, 15 Jun 2022 17:44:32 GMT

Check the route table that is GWLB endpoint is located, and make sure you have a route back to your internal resources (your VPCs CIDRs)

Re: Issues with Overlay Routing and AWS Gateway Load Balancer

A_Astardzhiev — Thu, 16 Jun 2022 08:28:56 GMT

Hi @Mandanajan,

Thank you for the suggestion, but I doubted the problem is in GWLBe route table. The reason for that is exact same setup (same GWLBe, same route table, everything works for East-West traffic the moment we have disabled route overlay. Also the outbound traffic works over the same GWLBe when overlay is enabled and I believe it wouldn't work if I was missing route for the VPC, right?

Re: Issues with Overlay Routing and AWS Gateway Load Balancer

justin.stone — Wed, 06 Jul 2022 18:40:43 GMT

Hi @A_Astardzhiev

I have the same problem, if I disabled Overlay then my east/west traffic worked fine, but outbound did not. With overlay on, it's the reverse. I tried 2.1.4, 2.1.6 and 2.1.7 plugins no change. I am also running 10.1.6. I just downgraded to 10.1.5h1 and now it all works, maybe give that a shot.

Re: Issues with Overlay Routing and AWS Gateway Load Balancer

A_Astardzhiev — Thu, 07 Jul 2022 09:06:42 GMT

Hi @justin.stone ,

That is intersting. At least I know I am not insane...

I think I had tried with 10.1.5...but can't remember if it was .5 or .5h1. Thanks I will give it a try.

Re: Issues with Overlay Routing and AWS Gateway Load Balancer

Centogene — Tue, 12 Jul 2022 16:13:28 GMT

We also: we have tried version 10.2.1 and downgraded to 10.1.6-h3 on AWS support advise, none work so far in.

We have not disable overlay routing yet. But will first try 10.1.5h1 as a suggestion and take a look.

Re: Issues with Overlay Routing and AWS Gateway Load Balancer

npandey — Fri, 15 Jul 2022 06:06:18 GMT

This is due to an existing bug which the team is actively working on. 10.1.5-h5 does not have this issue.

Thanks,

Nidhi

Re: Issues with Overlay Routing and AWS Gateway Load Balancer

Centogene — Fri, 15 Jul 2022 08:19:44 GMT

With 1 post to your name to say something is being fixed, and with all due respect: how do you know 'the team' (assume you mean Palo Alto dev) are actively working on it?

Can you provide more detail please @npandey

As we need an upgrade path into version 10.2 and beyond, and this bug (that is known) has not been fixed any any releases beyond 10.1.5-h5

Re: Issues with Overlay Routing and AWS Gateway Load Balancer

npandey — Fri, 15 Jul 2022 09:02:51 GMT

I have come across this issue and the reason I got tagged to this query. I have already raised this issue with the product team and that’s how I am aware about the Dev team looking into it.

If this is urgent and the customer is ok with NAT gateway, this could be a workaround otherwise we may have to wait for the fix to be officially available. If the customer needs a more official statement, please raise a TAC case.

Re: Issues with Overlay Routing and AWS Gateway Load Balancer

JHall15 — Mon, 18 Jul 2022 17:19:59 GMT

What was the outcome of downgrading to 10.1.5h1? I too am running 10.1.6-h3 and I have been banging my **bleep** head over this. I have a firewall pair in another AWS region running 10.0.7 and this works perfectly but not in 10.1.6-h3 in my AWS region where I need it to work.

Re: Issues with Overlay Routing and AWS Gateway Load Balancer

justin.stone — Mon, 18 Jul 2022 17:46:06 GMT

FWIW I opened a tac case few weeks ago, and they confirmed that 10.1.6 had issues with gwlb, as well as 10.2.2

Re: Issues with Overlay Routing and AWS Gateway Load Balancer

Centogene — Tue, 19 Jul 2022 11:20:27 GMT

Hello @JHall15

We went to 10.1.5-h5 and indeed this is the only Pan-OS revision that works. So basically, if this is in production you have limited options that put your environment at risk due to the out of date firmware.

I am opening up a TAC case as we need to add more weight to the issue to get this fixed.

Re: Issues with Overlay Routing and AWS Gateway Load Balancer

npandey — Tue, 19 Jul 2022 16:46:31 GMT

Hello,

Updating this post - The fix for the issue is committed to 10.1.7 version.

Thanks,

Nidhi

Re: Issues with Overlay Routing and AWS Gateway Load Balancer

mohamedridha — Tue, 26 Jul 2022 23:22:35 GMT

Just to say this issue has impacted me too after an upgrade to 10.2.2

It would be good to have a bug ID associate to this with some more information for what the root cause.

so far TAC only mention 10.1.7

Re: Issues with Overlay Routing and AWS Gateway Load Balancer

npandey — Wed, 27 Jul 2022 04:33:59 GMT

We have the fix for the issue integrated in 10.1.7. do you still see the issue ?

Re: Issues with Overlay Routing and AWS Gateway Load Balancer

mohamedridha — Wed, 27 Jul 2022 07:54:05 GMT

I don't see it released?

Re: Issues with Overlay Routing and AWS Gateway Load Balancer

Centogene — Thu, 28 Jul 2022 10:36:47 GMT

@npandey When is it released into the 10.2.x streams? 10.1.7 is out-of-date...

Re: Issues with Overlay Routing and AWS Gateway Load Balancer

JHall15 — Thu, 28 Jul 2022 12:44:03 GMT

I still don't see any indication 10.1.7 has been released let alone anything new in 10.2.x.

Re: Issues with Overlay Routing and AWS Gateway Load Balancer

Mandanajan — Fri, 05 Aug 2022 20:24:52 GMT

Just another thing to check, make sure you have enabled the Appliance Mode feature on the Transit Gateway, this feature ensures symmetric bidirectional traffic forwarding between VPC attachments. In other words, the forward and reverse flows are sent to the same firewall instance the same AZ for the lifetime of that flow. This allows firewalls to see both directions of the given flow thereby maintaining stateful traffic inspection capability inside Appliance VPC.

If the EC2 instances are deployed in two different VPCs in two different AZs. When the instances try to communicate with each other through AWS Transit Gateway with VPC attachments that are not in the same AZs results in asymmetric routing of packets. Forward flow and reverse flows from the same two communicating nodes, go to two different firewall instances in two different AZs, and the traffic is disrupted. This happens because, by default, when traffic is routed between VPC attachments, AWS Transit Gateway keeps the traffic in the same AZ where it originated until it reaches its destination.

Re: Issues with Overlay Routing and AWS Gateway Load Balancer

Mandanajan — Fri, 05 Aug 2022 20:31:45 GMT

Appliance Mode is disabled by default on the VPC attachments in AWS Transit Gateway. For VPC-to-VPC traffic inspection through a Security VPC, you are required to enable Appliance Mode on the VPC attachment connected to the Security VPC. However, enabling Appliance Mode is optional for inspection of traffic originating from a spoke VPC destined to the Internet via dedicated Egress VPC.