PA with ELB and ILB in Azure

[Ansh.mi]

We have the below setup:

internet->ELB(public ip)->VM Series(2)-> ILB->Web Servers

Where the 2 VM series Firewall in backend pool of both ELB and ILB, the issue here is the Health probe IP for both ELB and ILB is 168.63.129.16 so health probes always fails for one of them, I can resolve this issue by happing 2 VR.

But my question is I had similar setup on other firewalls (Checkpoint, Fortinet) it works fine with One Routing Instance.But on PA it does not work with one VR.

As of now I have route of  168.63.129.16 towards external interface, Health probes works fine for ELB. But in case of ILB PA is building 2 sessions and we are never seeing ack back from 168.63.129.16.The reason of building two sessions is that the first session is in INIT  state.

== 2018-11-21 21:10:34.174 -0800 ==
Packet received at fastpath stage, tag 125240, type ATOMIC
Packet info: len 66 port 17 interface 17 vsys 1
  wqe index 10461 packet 0x0xe05900bb00, HA: 0
Packet decoded dump:
L2:     12:34:56:78:9a:bc->00:0d:3a:30:78:75, type 0x0800
IP:     168.63.129.16->10.19.2.4, protocol 6
        version 4, ihl 5, tos 0x02, len 52,
        id 26606, frag_off 0x4000, ttl 128, checksum 27997(0x5d6d)
TCP:    sport 50255, dport 80, seq 3507572149, ack 0,
        reserved 0, offset 8, window 8192, checksum 13719,
        flags 0xc2 ( SYN), urgent data 0, l4 data len 0
TCP option:
00000000: 02 04 05 a0 01 03 03 08  01 01 04 02                ........ ....
Flow fastpath, session 125240 (set work 0xe054c8d600 exclude_video 0 from sp 0xe149818200 exclude_video 0)
IP checksum valid
* Dos Profile NULL (NO) Index (0/0) *
* Dos Profile NULL (NO) Index (0/0) *
Syn Cookie: pan_reass(new syn): c2s:0 c2s:nxtseq 3507572150 c2s:startseq 3507572150 c2s:win 14600 c2s:st 3 c2s:newsyn 0 :: s2c:nxtseq 2361026633 s2c:startseq 2361026633
 s2c:win 8192 s2c:st 1 s2c:newsyn 0 ack 0 nosyn 0 plen 0
Forwarding lookup, ingress interface 17
L3 mode, virtual-router 1
Route lookup in virtual-router 1, IP 10.19.2.4
Host route found, forward packet to host
Host service check passed, transmit to control plane

== 2018-11-21 21:10:34.174 -0800 ==
Packet received at ingress stage, tag 0, type ORDERED
Packet info: len 66 port 17 interface 17 vsys 1
  wqe index 10461 packet 0x0xe05900bb04, HA: 0
Packet decoded dump:
L2:     00:0d:3a:30:78:75->00:70:76:69:66:00, type 0x0800
IP:     10.19.2.4->168.63.129.16, protocol 6
        version 4, ihl 5, tos 0x00, len 52,
        id 0, frag_off 0x4000, ttl 64, checksum 24069(0x55e)
TCP:    sport 80, dport 50255, seq 2361026632, ack 3507572150,
        reserved 0, offset 8, window 14600, checksum 10280,
        flags 0x12 ( SYN ACK), urgent data 0, l4 data len 0
TCP option:
00000000: 02 04 05 b4 01 01 04 02  01 03 03 07                ........ ....

The similar setup works fine on other firewall where I am seeing 3 way handshake compelted  and only one session getting build. Now Is it the expected behaviour or some thing we can do to get this resolved?

[jperry1]

You have it right in terms of needing 2 VR's to route each health check for your ELB and ILB. you IP probes from the LB's you both come from two separate 168.x.x.x addresses. you have to use a separate VR and static route for the ILB health probe in order for the ILB to work. 

But i'm a bit confused because the VM-Series should only be in the backend pool of the ELB  and the Web servers should be in the backend pool of the ILB. The VM-series should route traffic out of the trust side to the ILB Virtual IP address unless you have a specific reason not to.

[Ansh.mi]

@jperry1thanks for your reply. Couple of points here:

1. IP for health-probe for both ELB and ILB is same.

2.In our setup we have an ILB where the the PA is in backend pool  so that traffic initiating from Azure VM can be send to the Fire

---

@jperry1 wrote:

You have it right in terms of needing 2 VR's to route each health check for your ELB and ILB. you IP probes from the LB's you both come from two separate 168.x.x.x addresses. you have to use a separate VR and static route for the ILB health probe in order for the ILB to work. 

But i'm a bit confused because the VM-Series should only be in the backend pool of the ELB  and the Web servers should be in the backend pool of the ILB. The VM-series should route traffic out of the trust side to the ILB Virtual IP address unless you have a specific reason not to.

---

wall.

My question now will this setup work with one VR on PA because in other firewalls it does work.

[jperry1]

Oh ok got it. so you must be using the Azure Standard Load Blancer as the ILB. if not I would suggest this. The reason that you can't use one VR is because of the way we route. The ILB health check needs to go back out of the internet the ILB health check comes but it will take the default route which is more than likely on a different interface.  To overcome this you have to have 2 VR's and place the interfaces in each VR. 

[Ansh.mi]

I got that part. I currently have one VR setup where a default route from external interface and have assymetric routing allowed on  on Palo Alto, so for ILB health proble syn comes on Internal Interface and syn-ack initiates from internal interface and checks the routing and moves out from the external Interface. So for some reason PA is building two session one for Syn packets coming from ILB to PA and other for response with Syn-ACK.The setup works fine  with other Firewalls because we are getting ACK back from the LB.

Note: Session with Syn Packet from ILB to PA is in INIT state, so I am also looking for a possible reason behind it.

[jperry1]

@Ansh.mi 

This issue is a fundamental routing issue not a VM-Series issue. Although you are experiencing this due to using ELB and ILB in Azure this would also happen in physical firewall based on the way the PANW firewall handles routing. If this is a bahavior you are interesting in requesting to change please reach out to your account team and regarding this and they can file a feature request. Thanks for the input and contributing in the AWS/Azure Discussions board. 

[sagar.debnath]

I am also facing the same issue. The trusted interface is configured as DHCP there there is no issue with Health probe I can see session getting completed in session browser, but while I switch the trust interface to static IP (10.205.96.5/24) the heath probe traffic is in INIT state.