Perfomance on PA 4060 - Huge Disappointment

[usvi]

We are having a poc at an ISP with a 4060 . It is a huge disppointment performance wise . We are seeing very high dataplane utilization on very small sessions. 50% on 220,000 sessions. 38% on 149,000 sessions. This is a box that is supposedly can handle 2M sessions .

Palo Alto is claiming there is nothing wrong with the unit as it is performing as expected based on the traffic . I thought PAN isn't a UTM . It is looking more like one to me .

[mikand]

The utilization on its own is pretty much couldnt care less since you will have the virtually the same latency with one packet vs 100.000 packets (until it hits the roof).

As you can see with your figures using 149k sessions yielded 38% utilization on the fpga's bringing you a ratio of 3921 sessions per percent.

With 220k sessions you had 50% which brings you a ratio of 4400 sessions per percent.

According to some tests performed by NSS Labs in autumn 2010 and spring 2011 the tested PA unit (4020) yielded higher true throughput than stated in the datasheets:

http://www.paloaltonetworks.com/literature/research/NSS-Labs-Report.pdf
http://www.paloaltonetworks.com/literature/research/NSS-Labs-Report-2011.pdf

So the question here is how you have setup the test-network along with which settings you use in your PA unit?

Are you for example having extensive use of NAT's and PBF's and stuff like that?

[usvi]

Nopes. Basic VW install no NAT or PBR . Simple network setup . Two zones trust and untrust .As an ISP , all apps are allowed in and out. At this point , I don't really care about NSS report . This is really world traffic not some in a lab . Our fear is when traffic really cranks up to about 1M sessions  in a few days , the box would die .

[mikand]

Is it possible for you to attach the current configuration?

Also which version are you using, 4.1.1 ?

And what does the internal counters of PAN says when you login to it using SSH?

Perhaps its your tests that produces shitloads of stalled sessions (just a thought) since NSS have their tests verified (just assuming but NSS's tests can be verified since they use IMIX and other tests if I remember correctly even if they are produced in a lab).

[usvi]

The PA 4060 is live on the ISP network b/w their routers and current FW s. THIS ISN'T A LAB OR SIMULATED TRAFFIC.

The boxes are running 4.1.1 . I dont have the running conf but can send the lastest tech support and AVR files if you wish .

[mikand]

Yes this would be interresting to see how the box is setup (you can remove stuff like ip addresses and such - more interrested in if and how any NAT; PBF, QoS, IPS, Antivirus, Anti Spyware, URL-categories, Security Policies etc are setup).

Also did you file this as a potential bug report to your support (because it would be interresting to get an explanation to potential max usage even if it didnt hit the roof as of yet)?

[usvi]

Where do I submit the potential bug report ?

[mikand]

Contact your service provider to file this as a bug.

You can also upload files to this forum (check "Attach files" input box below the box where you write your message - max size is today 50 MB, all file types allowed (but use txt or such preferly over doc etc)) but note that this forum is just for discussion and not support cases (meaning you should send this to your service provider in order to make it an official support case).

[usvi]

I opened a support case . Hopefully , they can figure out it bf traffic gets to a million session and the box is pulled out of the ISP's network .

[mikand]

Dont forget to keep this thread updated once you get some sort of result from the support.

[mikand]

Any answers from the support yet?

[usvi]

Sorry been busy . Well the box is holding . We are now averaging 600000 sessions and dataplane utilization is averaging 78% . We had another issue with very high utilization on the management cpu . It turns out we were logging 7000 records per second . Which was taxing the management cpu to the max . We had to play around with what we were logging and it has come down . The good thing is the box held with the 300% increase in traffic . That was our greatest fear and the POC is looking good.

Thanks

[NGS_SOC]

Maybe 5000 series suit better for your scenario... ssd tecnology helps in case of high log rate, better dataplane can handle better huge data traffic etc.

[mikand]

Would be interresting if you can do pings and traceroutes (both ICMP and UDP/TCP) over time through the PAN to see if you notice any raise in latency when your traffic goes up (and report back to this thread 🙂

Also did you update to 4.1.2?

[usvi]

Upgraded to 4.1.2 and did a load balancing with two PA 4060s and it has helped with the performance issue .