Aggregate Interface Throughput limit - Multi VSYS - Shared Gateway.

[mcnairi]

Hi Community

I have multiple VSYS setup that also uses Shared Gateway for collating access to my Data Centre to and from each VSYS.

I have a PA5250 setup running OSPF with a 40G routed connection to my Data Cente (Northbound) - in the shared gateway area on a dedicated P2P 40G interface..

Each VSYS has a secure zone and an unsecure zone. The Unsecure Zone is the shared gateway zone (standard SG setup) and the secure zone is a southbound facing interface within that VSYS that we apply policy on.

Southbound the Secure Zone is serviced using an Aggregate Interface made up of 4 x 10G physical interfaces on the Firewall.

In this Aggregate Interface I have logical sub interfaces all using different layer 2 tags each "secure" network below.

So for Example VSYS2 "Company A" has a secure area zone with sub interface AE4.101, and an interface in the unsecure zone using the Shared Gateway.

When sending traffic from AE4.101 to AE4.102 I hit a limit of 1.06Gbps and cannot exceed with any single TCP stream.

Further more, when I generate UDP traffic over 1.1Gbps, this destroys the OSPF adjacancies across the entire Firewall, even on interfaces that are not related to the flow.

Seems to me like some kind of logical limitation ( in built QOS, Ddos or control plane issue) with servicing traffic originating from and going to the same AE interface when using sub interfaces.

Anyone heard of this or experienced any similar issues?

Traffic going through the firewall either from the Shared Gateway to Sub interface or vice versa is fine at 10Gbps, its just sub interface to sub interface Im seeing the issue.

Thanks everyone in advance...:)

[nextgenhappines]

That is because the inter vsys traffic is not process by the offload processor.  

https://live.paloaltonetworks.com/t5/Learning-Articles/Differences-between-packets-in-slow-path-fast...

you can also check while the traffic is passing the firewall, go to cli and show session id xxxx, you will see the traffic is not getting offload. 

One way to work around this, create multiple dedicated untrust interface and virtual router per vsys, and peer each of vsys directly to the upstream router(s).   

[mcnairi]

Thanks Nextgenhappiness.

Can I just say that I have had a TAC case open for 3 weeks and your the only person that has confirmed my suspicion that the firewall is not handling the traffic as it should.

Did not know that Inter-Vsys Traffic is not offloaded....WOW - major performance difference.

Really grateful for your input. The article is really interesting thank you:

Very interested in your work around, - I dont really understand what you mean though, could you elaborate a little>

Took the caveat list from the web page you provided:

There is also a type of traffic that will only be processed by CPU and will never be offloaded.

• ARP (and all other non-IP traffic)
• IPSec
• Decrypted sessions
• VPN sessions
• non-TCP/UDP
• Firewall bound session
• Inter-vsys sessions
• PBF session without next hop
• NAT64

[mcnairi]

Would this also mean that I could not utilise the Shared Gateway? (I dont have many many interfaces to use on the firewall)

Would that also mean that all Inter - Vsys traffic would route via the upstream Router, not across the firewall shared gateway?

The whole point of my design is that the Shared Gateway means I dont have to have mulitple, VR's using dedicated seperate uplink interfaces to my Data centre?

[mcnairi]

Hi Nextgen

Here is an output of an inter-vsys session that is using iperf3 reaching 1.06gbps.

where is the confirmation that its not being offloaded?

admin@GED-HRD-PA5250(active-secondary)> show session id 33591197

Session 33591197

c2s flow:
source: 10.64.3.20 [ECOM-LIVE-2-PROD-LIVE]
dst: 10.68.1.20
proto: 6
sport: 49715 dport: 5201
state: ACTIVE type: FLOW
src user: unknown
dst user: unknown
ecmp id: 8000

s2c flow:
source: 10.68.1.20 [VSYS-ECOM-ZONE-LIVE]
dst: 10.64.3.20
proto: 6
sport: 5201 dport: 49715
state: ACTIVE type: FLOW
src user: unknown
dst user: unknown
ecmp id: 8000

Slot : 1
DP : 1
index(local): : 36765
start time : Tue Aug 7 10:30:29 2018
timeout : 3600 sec
time to live : 3371 sec
total byte count(c2s) : 644
total byte count(s2c) : 390
layer7 packet count(c2s) : 8
layer7 packet count(s2c) : 6
vsys : vsys6
application : iperf
rule : ECOM-2-PROD-ANY
session to be logged at end : True
session in session ager : True
session updated by HA peer : False
session owner is HA A/A local device : True
session setup locally HA A/A : True
layer7 processing : completed
URL filtering enabled : False
session via syn-cookies : False
session terminated on host : False
session traverses tunnel : False
captive portal session : False
ingress interface : ae4.201
egress interface : ae4.205
session QoS rule : N/A (class 4)
tracker stage l7proc : ctd decoder bypass
end-reason : unknown

[nextgenhappines]

@mcnairiIt will depend on how many vsys you have. You can keep the shared gateway design, only those vsys that has performance issue, remove those vsys from the shared gateway and create their own untrust interface/zone and dedicated VR.  You could use sub interface as your uplink interfaces to your data center.  You will need to burn more IP addresses and few more vlans

If the session is being process by the offloader, in the session id output "offload: yes" under c2s flow and s2c flow after the dst user:

[mcnairi]

Hi Nextgen

Appreciate your help on this, have advised TAC of your findings, amd waiting for an answer from them.

However I found the above link which talks about how you identity if traffic is offloaded:

https://live.paloaltonetworks.com/t5/Management-Articles/Firewall-offloading-traffic-how-to-disable/...

Basically if the output says ctd decoder bypass - it is being offloaded.

In my session output further up the thread, the session is inter-vsys and it is being offloaded: 

ctd decoder bypass

So, if Inter-Vysys traffic is being accelerated, does that mean its a different issue casuing the throuughput limitation.

Ps, I tried Interface 1 to Interface 2 on the same Vysys and got way above 1Gbps using iperf.

[nextgenhappines]

Looking for application SSL that goes from your trust zone to untrust zone ( to the Internet) and show session id xxxxx you should see the offload: yes there..

All the sessions between intervsys using shared gateway will not get process by the offloader.

[mcnairi]

Hi.

I agree with you, however whats confusing me is that the traffic output i posted above - is - Inter-Vsys.

Interface AE4.101 is in a different Vsys to AE4.105 (different sec zones too)

And the ouput says it is offloaded.

So does this mean its not actually offloaded - because performance indications Im getting 1.06bgps, but the session output says it is offloaded....

[nextgenhappines]

hi,

that session you posted is not offloaded.  before you start the inter vsys iperf test, check your dp usage, show running resources min last 5  start your iperf test for at 5 minutes or longer, while the iperf test is running , check the dp usage again, if you see all the dp cores usage goes up and verify which dp is running high, if that match your iperf session.  that tells you if the session is offloaded or not.  you can repeat the same iperf test behind the firewall and behind the upstream router and check the session log and dp usage while iperf test is running.  in the session id outout, you should see offload: yes.

hope this helps.,