AWS VM Series Gateway Load Balancers not working

[craig.beamish]

Hi All

Has anyone else had a play with the GWLB on AWS?
I know it must be PAN-OS 10.0.2 or higher to work,
I have tested with multiple instances, 
As a bump in the wire it works fine. until you apply NAT, then it doesn't work at all for any traffic that is NAT'd. 
I have an open TAC for this, they are replicating the fault to work it out but surely this was all tested before it went public.

I also found overlay routing breaks traffic flow. its not documented anywhere that I could find but what I found was it processes the GENEVE traffic in the virtual router where without it, is just an in-return non routed flow. 

If you've tinkered with it and actually got inbound/outbound NAT and/or overlay routing to function, please let me know what you did. 

sadly the documentation just doesnt provide any decent clarity for this feature.

Also extremely disappointed they havent integrated this into version 9.1.
I am hopeful they will add it with 9.1.7 in a functional state as I am not planning to move my clients to 10.0 until the list of known issues is about 1/4 its current size.

[rapatil]

Hi Craig,

Thanks for your feedback. 

How are you deploying the GWLB with VM-Series? Are you using any of the templates provided on our Github repo? 

https://github.com/PaloAltoNetworks/AWS-GWLB-VMSeries

If you are using a CFT with an autoscale template, then it will create a NAT GW along with other components. The template also takes care of the automatic route population for any new APP VPCs. 

Thanks,

Raj

[jmeurer]

As of this time, break out routing is not supported.  The traffic must stay in the Geneve tunnel.  In reading this, it appears you are addressing the outbound use case.  In that traffic flow, the NatGW must be used as the next hop beyond the GWLBe as depicted in this flow.  

[craig.beamish]

We are using a nat gateway. outbound works just fine until you apply NAT of any sort. just trying to apply any sort of NAT to change the direction of the traffic breaks it. 

so if i put a nat rule that traffic to 1.1.1.1 gets d-nat to  8.8.8.8, the traffic never exits the firewall.
key use for this was for inbound traffic, redirecting inbound traffic to the correct ALB  that lives in another vpc, the traffic seems to get dropped at the firewall, even though pcaps show it *thinks* it is being forwarded on, its not.

[jmeurer]

Inbound requires ingress routing to use the GWLB without SNAT.  You can do that within the application vpc using a public-facing LB in front of the application.

Or if you want to have a dedicated inbound VPC, you use the same design as above but move your pool members across the TGW.

If you prefer the traditional Load Balancer sandwich design where the firewalls are pool members of the front door LB and you are going to SNAT/DNAT to the application, you would either use a dedicated set of firewalls or add new Untrust and Trust interfaces to the firewall as ETH3/4 and use those for ingress outside of the GWLB.  This is necessary as the GWLB traffic must hairpin inside of the Geneve tunnel, you cannot insert flows into the tunnel from another interface.  

[craig.beamish]

neither of those suit the application Im doing.

We have multiple inbound services that sit behind the firewall. 
the current layout is the trust/untrust sandwich but we would prefer to move away from the NLB design as they have a capacity limitation in regards to autoscale groups. 

the design plan is that we have 'anchor' network addresses for each inbound service, the traffic comes in via the IGW, steered through the geneve tunnel to the palo, at which point we apply a destination NAT to the actual application load balancer of that service (which is in a different vpc), the traffic is still supposed to egress the geneve tunnel just with a different destination address, but the palo seems to drop it (even though the palo pcap believes it is forwarded on (seen in transmit stage with correct destination IP) 

on your comment re: overlay routing, why did they put the feature command in there if they havent got it working yet? /logic 😕

[jmeurer]

At this point, I would suggest you reach out to your Account Team to engage with one of the Consulting Engineers to discuss over zoom and whiteboard.  There is only so much we can accomplish in the message board and a further understanding of your flows is warranted.

[craig.beamish]

For those playing at home,

In further discussions with @jmeurer, AWS apply a 5 tuple hash on the traffic to ensure return path, so applying a NAT breaks the traffic flow and AWS drops the traffic.

Overlay routing is not yet functional, hopefully it will be in 10.0.4 or 10.0.5 and I can test if I can get the NAT to work for me in that aspect. I am curious if the return traffic would exit via the same geneve tunnel given theres no routes through it in that situation, only time will tell.

[Sandeep_Darak]

Did this issue Fixed.

I am also facing challenges with AWS GWLB. Traffic is Sourced from Outside towards inside.
Traffic monitor is showing traffic from Outside to Outside.

Not sure why.....

[tostern]

Hi @Sandeep_Darak 

that in depends how you setup your interfaces. when you have only one interface then is of course your traffic recognized als interzone fraffic to Outside -> Outside. when you want to split it then you have to create sub interfaces and map them to a another zone and adopt your FW VR routing.

[Sandeep_Darak]

Hi @tostern 

In my case my setup is with two interface Eth1/2 (Inside) & Eth1/3(Outside). So whenever I hit traffic from Outside server to Inside Server traffic logs should say Traffic from Out to In. which is not happening. 

[pkhavkine]

I had a 3 interface setup working: GENEVE In/Out through eth1/1, then into eth1/2 -> NAT -> out of eth 1/3 to the ouetside.

Traffic would end up passing through the firewall twice.

On the other hand GWLB seems to break GP, so cannot run GP portal/Gateway on the outside interface.

[Sandeep_Darak]

My design is as per below. Let me know if any issue.

Server-1 (Outside)==>TGW==>SecurityVPC==>GWLBe==>EndPoint Service==>GWLB==>PaloAlto Outside interface (Eth1/1)==>Pa Processing==>PaloAlto Inside interface(Eth1/2)==> Server-2 (Inside).

I am not using GP instead traffic is ping/ssh. Whenever i process the traffic from Outside to Inside traffic logs saying traffic outside to outside hence not matching correct policy and not processing.  

[jmeurer]

At this time, GWLB deployments do not support routing outside of the GENEVE interface.  The traffic must hairpin back to the GWLB.

Also, there is a known issue with GP not working on a GWLB enabled firewall that will be resolved in a future release.

[pkhavkine]

Thanks for letting me know that it's a known issue with GP, any indication on when to expect a fix?

Thanks