After Upgrading our PA-820 to 11.0.2, we're seeing lots of data on dns-base application.

[RaulTrujillo]

After Upgrading our PA-820 to 11.0.2, we're seeing lots of data on "dns-base" application.  In a 24-hour period, I'm seeing 5PBs+ of data coming through, which is way over our limits for our internal network (two DNS servers at 1GB NIC each) and external network.  Our ISP bandwidth is 500/500 Mbps.  So it's going way over our MAX capacity.  However, if you look at the Management Plane and the Data Plane, they are both under 25%.  I'm thinking it's just not showing the numbers properly.  The DNS Servers that supposedly are communicating with external DNS Servers (8.8.8.8, 1.1.1.1, OpenDNS, etc.) are not showing tons of traffic.

[OtakarKlier]

Hello,

Where is the traffic going, externally? If yes then it could be data exfiltration. Please research the traffic and see it its a treat actor. If you are only allowing DNS traffic to those several IP's, block all the rest.

Here is a short video on securing DNS.

https://www.youtube.com/watch?v=ROIAYSEbTuo

Regards,

[RaulTrujillo]

Thanks for your reply.  I did add the recommended DNS security policy.  I am a bit more secure.  Thank you.  However, I'm still seeing TBs of data hitting my "dns-base."  I could see that in Monitor -> App Scope -> Network Monitor, the bandwidth is capped at 281.47 TBs 15 minutes apart.  This bandwidth is supposedly going out from my internal DNS Server(s) 10.0.0.6 and 10.0.0.9, to an external DNS Server (1.1.1.1 or 8.8.8.8).  However, with a 500/500 Mbps, I doubt I would hit that many TBs in 15-minute increments.

[OtakarKlier]

Hello,

That does seem suspicious. I would open a support case and see if there might be a new bug in the code.

Regards,

[RaulTrujillo]

I just got a reply from one of the engineers.  He said that there are other similar cases.  They are currently investigating the issue.

Regards,

Raul Trujillo

[RudolfTorvenyi1]

Hello, I see the same on these FWs and PANOS version

PA-820 10.1.8

PA-440 11.0.2

PA-850 10.2.4-h3

In the traffic log using this filter "( app eq 'dns-base' ) and ( bytes geq '12302' )" I see some sessions that are 281T big 

I have also a case opened from 13.7.2023, but I didn't get any response from the support yet.

Regards,

Rudolf

[BenjaminSutterSKAN]

I have the same issue for a PA-440 11.0.2 firewall.

In 1h 281.5TB of dns data... to external dns server.

[RaulTrujillo]

I have a second higher-up engineer who is starting to look at my issue.  My case # is 02618638.  The first engineer ran multiple tests and collected lots of data.  By the way, I'm going to share other info that I have on my network:

Switches:  Junipers

Access Points:  Ruckus (Cloud Managed)

ISP Provider:  AT&T Fiber

I noticed Rudolf is getting the same issue but with other firmware versions.  I began to see this after upgrading to 11.x.x.  Unless I had missed it.

[RaulTrujillo]

Great observation Rudolf.  I see the same thing when I use:

( app eq 'dns-base' ) and ( bytes geq '150000' )

Most of that data is 281.5T.  So, now, what I'm guessing is that "one" session takes up the 281.5T of fictitious data.  Here are more details about the data:

Type end
Bytes 281474976579783
Bytes Received 2147483647
Bytes Sent 199
Repeat Count 1
Packets 1
Packets Received 4294967295
Packets Sent 2

Regards,

Raul Trujillo

[mlinsemier]

We are still running into the dns-base 281.5T session issue on PAN-OS 10.2.5 and PAN-OS 11.0.2-h1.  I am not certain if it started on PAN-OS 10.2.4 or a later hotfix (h2,h3,h4) but wasn't present in PAN-OS 10.2.3-hx. The same information is being exported via Netflow as well so its certainly in the logs and not just cosmetic to the GUI.  That being said the amount of traffic is wrong as we have validated it via packet captures (I wish I could do 281.5T in a 2 minute session)

My next step was to dig through the apps/threats updates for the last month or so to see if there were any changes to the DNS decoder as perhaps its not a PAN-OS bug. 

I have opened TAC Case#: 02672022 on this which is currently being investigated.  

[RaulTrujillo]

Right.

You can also refer them to my case:  My case # is 02618638

However, it appears to be an issue with "dns-base" and not necessarily 11.x.  That is when I noticed it.  After looking through Rudolf's filter, it makes more sense that it's an app (dns-base) issue.

[brycejett]

On 10.2.5 we see same issue. 281.5T bytes for 30s DNS connection. Bytes transmitted 178(correct confirmed in capture). Bytes received 2,147,483,647 (no response in capture). And somehow the total bytes add up to 281,474,976,579,762. Capture did not have any indication of a response this big and the DNS server was our internal server. 

[RaulTrujillo]

Hi everyone,

The engineer that is my the case mentioned above just replied the following:

------- snip ----- start here
Greetings!

I hope you are doing good.

Thank you for your patience. The fix has been coded in 11.0.4 and 10.2.8 PAN-OS versions. Both versions do not have an ETA as of now. But you can track down the issue with this issue ID PAN-227639 in the release notes once published.


Meanwhile, If you have any queries feel free to reach out to me, I will be glad to assist you.

Have a nice day!

----- snip -- end here ----

[WalidSleiman]

Hi Raul,

Thank you for your updates! I've been facing the same issue on PA-5220 which was on version 10.2.4-h3 and the same issue remained even after upgrading to 10.2.5.

Please can you share with us any updates on this bug PAN-227639 provided by Palo Alto TAC support on your support case and the public URL link for that bug id as I can't find it on the web.

Thank you in advance.

Best regards

[RaulTrujillo]

Hi Walid,

The CASE #the engineer used was:  CASE 02618638

I submitted the case on 7/5/2023.

The latest response to the case is the following:

Hi Raul,

Greetings!

I hope you are doing good.

The fix has been coded in 11.0.4 and 10.2.6 PAN-OS versions. Both versions do not have an ETA as of now. But you can track down the issue with this issue ID PAN-227639 in the release notes once published. Could you let me know whether we can proceed with this case toward soft closure with your permission? Also, you can reopen the case if you have any queries or doubts regarding this issue.

Meanwhile, If you have any queries feel free to reach out to me, I will be glad to assist you.

Have a nice day!