Clarity on Overlay routing with GWLB for Combined (Centralized Egress + Distributed Ingress) deployment model

[Yelzamani]

Hi, 

I am looking for some clarity on the Overlay routing feature on VM Series FW. 

I am using the Combined (Centralized Egress + Distributed Ingress) deployment model described in this Securing Application in AWS - Centralized Model Deployment Guide. 

In this pattern, overlay routing is enabled so that the FW can handle both inspection and NAT for outbound traffic flows. It allows packets to leave the VM-Series firewall through a different interface than that which they entered through.

I have built this out in my lab and it works well but I want to understand why it does. 

Egress Traffic 

I understand for outbound flow (Traffic initiated from the VPC to the internet) - The traffic from the EC2 instance leaves the VPC via the TGW and lands on the TGW attachment in the Security VPC, which then sends it to the GWLB endpoint for inspection on the FW. With Overlay routing, the FW is able to check the destination address (internet bound) and directly route it out its public interface. 

EC2 --> TGW (App VPC) --> TGW (Security VPC) --> GWLBe --> (inside)VM Series FW(outside)---> IGW 

Ingress Traffic

Here is where it gets confusing. For inbound traffic (Traffic initiated from internet to VPC) 

Inbound flow

The traffic first hits the GWLB endpoint in the app VPC and gets send to the Inspection VPC for inspection, FW inspects (destination is app VPC) and sends the traffic back out the inside interface. Traffic gets send back through the GWLB endpoint and on to the ALB and to the EC2 instance 

Client --> IGW --> GWLBe(App VPC) --> GWLB (Inspection VPC) ---> (inside)VM Series FW (inside) --> GWLB (Inspection) ---> GWLBe (App VPC) ---> ALB (SNAT) ---> EC2 

Return flow 

Here the EC2 responds directly to the ALB private IP, ALB forwards to the internet via the GWLB endpoint, which then gets sent to the Inspection VPC for inspection, FW inspects and sends the traffic back out the inside interface to the GWLB and it makes it way back to GWLB endpoint in the App VOC and to the client on the internet.  

EC2 --> ALB (APP VPC) ---> GWLBe --> GWLB --> (Inside) VM series FW (inside) --> GWLB/GWLBe --> IGW ----> Client

With overlay routing, I expected the FW to see that the destination address on the return flow is internet bound and should have sent it out its outside/public interface (like it did on the outbound flow) - which would of course break the flow and cause asymmetric routing. 

How does the FW know to send the return flow back out its inside interface? 

I did not have to configure any PBRs to influence the routing. 

[mb_equate · Accepted solution]

I agree it does sound counterintuitive. I _think_ this is due to the firewall having already received the traffic from the GWLB on the inbound leg, matching the return flow to the same session (which in this case includes the TLV headers associated with the GENEVE encapsulation) and symmetrically returning the traffic to the GWLB. Reading the return traffic flow for the combined option in the deployment guide:

"The firewall receives the traffic from the GWLB on its private dataplane interface. The firewall receives the traffic and associates the return traffic to an active session on the firewall. The firewall uses the session information, including the TLVs associated with the Geneve encapsulation to return the traffic to the GWLB"

I find the documentation doesn't go into enough technical detail for those that want to understand how stuff works not just build it 😉

[Yelzamani]

Thank you @mb_equate 

Yes, that's what I was thinking as well and it makes sense.

So the reason I asked this question was a colleague also tested this out and in their case, the FW sent the return flow out its Public interface - causing asymmetric routing. I was not able to reproduce the issue though, so I wanted to check if it was a configurable option.

It is probably a misconfiguration/bug in their FW.