Netflow Bug in PAN OS 10.1.4-h4 and earlier OS 10 versions

[jeffeager]

This post is more of a survey of anyone that may be experiencing the same issue. We have a case open with PA and they have identified a bug. The fix, as I'm told, is that it will be addressed in a future release but they couldn't say when or what release. I searched the Live Community forums and found the issue occurring in an older 10.x version.  https://live.paloaltonetworks.com/t5/general-topics/change-in-netflow-behavior-in-pan-os-10/m-p/3910... . This earlier post is over a year old. Our issue is identical specifically, NF is reporting application traffic in the Tbps and even Pbps range since we upgraded from 9.1.4 to 10.1.4-h4. As we use PA native SDWAN this is impacting all reporting from our WAN sites, as well as DC traffic reporting, Internet traffic reporting, etc. Our netflow collectors (Scrutinizer and Riverbed Steel Central) have been rendered useless while PA decides when to release a fix. 

I'm just curious if anyone else has experienced this bug? If so, what has been your recourse to address the issue, or are you simply awaiting a fix as we are?

[MHamad]

Hello, yes, we faced this in PAN OS 10.1.6 as well. Could you please tell us what was the Bug ID they have mentioned?

[ScottySD]

I have had a ticket open with support for almost a year on this issue.  It took 6 months of back and forth before they acknowledged they could reproduce the issue.  I still don't have a bug ID.  

From support

We are finding that there are some scenarios in logrcvr netflow where the netflow buffer does not get reset. This may cause the buffer to potentially get sent out twice before getting reset.

This behavior will be addressed in the following PAN-OS versions: 11.1.0, 11.0.4, 10.1.12, 10.2.8

There is not ETA for 10.1.12 yet, but we know internally that 10.1.11 will be released at the end of September. This means that the 10.1.12 fix will be released in some months after September.

[ScottySD]

The bug ID is PAN-207003 and will be addressed in the following PAN-OS versions: 11.1.0, 11.0.4, 10.1.12, 10.2.8.

[jeffeager]

I don't know that your NF issue was the same as the one I originally posted above. However, we were advised of a fix 11/22/22 (We opened the case 3/22/22). NOTE: We did not go either 10.1.7 or 10.1.8. We waited and upgraded to 10.2.3-h4. 

Case Notes:

netflow issue is resolved and fix is available in 10.1.7 or latest PanOS version.

PAN-186891
Fixed an issue where NetFlow packets contained incorrect octet counts.
===
https://docs.paloaltonetworks.com/pan-os/10-1/pan-os-release-notes/pan-os-10-1-7-known-and-addressed...

PANOS 10.1.8 has been marked as preferred release which will have the fix as well.
https://live.paloaltonetworks.com/t5/customer-resources/support-pan-os-software-release-guidance/ta-...

[jeffeager]

Would you be willing to post the issues you are having with NF? Also, see my post below concerning our NF resolution.

[ScottySD]

We are seeing the throughput reported by netflow to be 2-5x what the traffic actually is.  For example, a 1Gbps interface reporting flow data @ 2Gbps.  We compare netflow against SNMP reporting of the interfaces and see this discrepancy.  We use scrutinizer and have also tested a variety of other collectors with the same results.  We are running PA460s and PA3420s.  Some on 10.1.10, 10.2.5, all platforms have the same issue.

[jeffeager]

Thanks for the response @ScottySD Your problem is almost identical to what we saw. We also tested against Scrutinizer, and other collectors. The only difference we saw is that NF reported a great deal more traffic than what you are seeing. I feel your pain on this, as it renders your NF collector useless for Palo Alto devices.