topic Re: After Upgrading our PA-820 to 11.0.2, we're seeing lots of data on dns-base application. in General Topics

After Upgrading our PA-820 to 11.0.2, we're seeing lots of data on dns-base application.

RaulTrujillo — Wed, 12 Jul 2023 17:41:50 GMT

After Upgrading our PA-820 to 11.0.2, we're seeing lots of data on "dns-base" application. In a 24-hour period, I'm seeing 5PBs+ of data coming through, which is way over our limits for our internal network (two DNS servers at 1GB NIC each) and external network. Our ISP bandwidth is 500/500 Mbps. So it's going way over our MAX capacity. However, if you look at the Management Plane and the Data Plane, they are both under 25%. I'm thinking it's just not showing the numbers properly. The DNS Servers that supposedly are communicating with external DNS Servers (8.8.8.8, 1.1.1.1, OpenDNS, etc.) are not showing tons of traffic.

Re: After Upgrading our PA-820 to 11.0.2, we're seeing lots of data on dns-base application.

OtakarKlier — Wed, 12 Jul 2023 20:31:52 GMT

Hello,

Where is the traffic going, externally? If yes then it could be data exfiltration. Please research the traffic and see it its a treat actor. If you are only allowing DNS traffic to those several IP's, block all the rest.

Here is a short video on securing DNS.

https://www.youtube.com/watch?v=ROIAYSEbTuo

Regards,

Re: After Upgrading our PA-820 to 11.0.2, we're seeing lots of data on dns-base application.

RaulTrujillo — Fri, 14 Jul 2023 13:24:57 GMT

Thanks for your reply. I did add the recommended DNS security policy. I am a bit more secure. Thank you. However, I'm still seeing TBs of data hitting my "dns-base." I could see that in Monitor -> App Scope -> Network Monitor, the bandwidth is capped at 281.47 TBs 15 minutes apart. This bandwidth is supposedly going out from my internal DNS Server(s) 10.0.0.6 and 10.0.0.9, to an external DNS Server (1.1.1.1 or 8.8.8.8). However, with a 500/500 Mbps, I doubt I would hit that many TBs in 15-minute increments.

Re: After Upgrading our PA-820 to 11.0.2, we're seeing lots of data on dns-base application.

OtakarKlier — Fri, 14 Jul 2023 14:25:18 GMT

Hello,

That does seem suspicious. I would open a support case and see if there might be a new bug in the code.

Regards,

Re: After Upgrading our PA-820 to 11.0.2, we're seeing lots of data on dns-base application.

RaulTrujillo — Thu, 20 Jul 2023 17:31:29 GMT

I just got a reply from one of the engineers. He said that there are other similar cases. They are currently investigating the issue.

Regards,

Raul Trujillo

Re: After Upgrading our PA-820 to 11.0.2, we're seeing lots of data on dns-base application.

RudolfTorvenyi1 — Tue, 01 Aug 2023 09:22:16 GMT

Hello, I see the same on these FWs and PANOS version

PA-820 10.1.8

PA-440 11.0.2

PA-850 10.2.4-h3

In the traffic log using this filter "( app eq 'dns-base' ) and ( bytes geq '12302' )" I see some sessions that are 281T big

I have also a case opened from 13.7.2023, but I didn't get any response from the support yet.

Regards,

Rudolf

Re: After Upgrading our PA-820 to 11.0.2, we're seeing lots of data on dns-base application.

BenjaminSutterSKAN — Wed, 09 Aug 2023 11:33:32 GMT

I have the same issue for a PA-440 11.0.2 firewall.

In 1h 281.5TB of dns data... to external dns server.

Re: After Upgrading our PA-820 to 11.0.2, we're seeing lots of data on dns-base application.

RaulTrujillo — Wed, 09 Aug 2023 12:56:08 GMT

I have a second higher-up engineer who is starting to look at my issue. My case # is 02618638. The first engineer ran multiple tests and collected lots of data. By the way, I'm going to share other info that I have on my network:

Switches: Junipers

Access Points: Ruckus (Cloud Managed)

ISP Provider: AT&T Fiber

I noticed Rudolf is getting the same issue but with other firmware versions. I began to see this after upgrading to 11.x.x. Unless I had missed it.

Re: After Upgrading our PA-820 to 11.0.2, we're seeing lots of data on dns-base application.

RaulTrujillo — Wed, 09 Aug 2023 13:14:09 GMT

Great observation Rudolf. I see the same thing when I use:

( app eq 'dns-base' ) and ( bytes geq '150000' )

Most of that data is 281.5T. So, now, what I'm guessing is that "one" session takes up the 281.5T of fictitious data. Here are more details about the data:

Type end
Bytes 281474976579783
Bytes Received 2147483647
Bytes Sent 199
Repeat Count 1
Packets 1
Packets Received 4294967295
Packets Sent 2

Regards,

Raul Trujillo

Re: After Upgrading our PA-820 to 11.0.2, we're seeing lots of data on dns-base application.

mlinsemier — Tue, 22 Aug 2023 19:15:35 GMT

We are still running into the dns-base 281.5T session issue on PAN-OS 10.2.5 and PAN-OS 11.0.2-h1. I am not certain if it started on PAN-OS 10.2.4 or a later hotfix (h2,h3,h4) but wasn't present in PAN-OS 10.2.3-hx. The same information is being exported via Netflow as well so its certainly in the logs and not just cosmetic to the GUI. That being said the amount of traffic is wrong as we have validated it via packet captures (I wish I could do 281.5T in a 2 minute session)

My next step was to dig through the apps/threats updates for the last month or so to see if there were any changes to the DNS decoder as perhaps its not a PAN-OS bug.

I have opened TAC Case#: 02672022 on this which is currently being investigated.

Re: After Upgrading our PA-820 to 11.0.2, we're seeing lots of data on dns-base application.

RaulTrujillo — Tue, 22 Aug 2023 20:24:41 GMT

Right.

You can also refer them to my case: My case # is 02618638

However, it appears to be an issue with "dns-base" and not necessarily 11.x. That is when I noticed it. After looking through Rudolf's filter, it makes more sense that it's an app (dns-base) issue.

Re: After Upgrading our PA-820 to 11.0.2, we're seeing lots of data on dns-base application.

brycejett — Fri, 01 Sep 2023 19:15:36 GMT

On 10.2.5 we see same issue. 281.5T bytes for 30s DNS connection. Bytes transmitted 178(correct confirmed in capture). Bytes received 2,147,483,647 (no response in capture). And somehow the total bytes add up to 281,474,976,579,762. Capture did not have any indication of a response this big and the DNS server was our internal server.

Re: After Upgrading our PA-820 to 11.0.2, we're seeing lots of data on dns-base application.

RaulTrujillo — Mon, 04 Sep 2023 19:05:44 GMT

Hi everyone,

The engineer that is my the case mentioned above just replied the following:

------- snip ----- start here
Greetings!I hope you are doing good.Thank you for your patience. The fix has been coded in 11.0.4 and 10.2.8 PAN-OS versions. Both versions do not have an ETA as of now. But you can track down the issue with this issue ID PAN-227639 in the release notes once published.Meanwhile, If you have any queries feel free to reach out to me, I will be glad to assist you.Have a nice day!

----- snip -- end here ----

Re: After Upgrading our PA-820 to 11.0.2, we're seeing lots of data on dns-base application.

WalidSleiman — Mon, 18 Sep 2023 12:21:19 GMT

Hi Raul,

Thank you for your updates! I've been facing the same issue on PA-5220 which was on version 10.2.4-h3 and the same issue remained even after upgrading to 10.2.5.

Please can you share with us any updates on this bug PAN-227639 provided by Palo Alto TAC support on your support case and the public URL link for that bug id as I can't find it on the web.

Thank you in advance.

Best regards

Re: After Upgrading our PA-820 to 11.0.2, we're seeing lots of data on dns-base application.

RaulTrujillo — Mon, 18 Sep 2023 19:50:05 GMT

Hi Walid,

The CASE #the engineer used was: CASE 02618638

I submitted the case on 7/5/2023.

The latest response to the case is the following:

Hi Raul,

Greetings!

I hope you are doing good.

The fix has been coded in 11.0.4 and 10.2.6 PAN-OS versions. Both versions do not have an ETA as of now. But you can track down the issue with this issue ID PAN-227639 in the release notes once published. Could you let me know whether we can proceed with this case toward soft closure with your permission? Also, you can reopen the case if you have any queries or doubts regarding this issue.

Meanwhile, If you have any queries feel free to reach out to me, I will be glad to assist you.

Have a nice day!

Re: After Upgrading our PA-820 to 11.0.2, we're seeing lots of data on dns-base application.

mlinsemier — Mon, 18 Sep 2023 20:27:31 GMT

If you opened a ticket with TAC can I ask that you escalate it with your account manager and sales engineer as this should be hotfixed in my opinion and my Palo Alto support team concur. This is causing havoc with our traffic anomaly security tools and while 10.2.6 will tentatively be released the the week of 9/25, 11.0.4 is at minimum 60 days out and we are forced to run this on our PAN-1410 appliances.

Re: After Upgrading our PA-820 to 11.0.2, we're seeing lots of data on dns-base application.

RaulTrujillo — Tue, 19 Sep 2023 15:57:20 GMT

Hi,

I just submitted the case to see if it can be escalated to a HotFix. The engineering tech I'm working with said he'll get back with me for an update.

Re: After Upgrading our PA-820 to 11.0.2, we're seeing lots of data on dns-base application.

RaulTrujillo — Wed, 20 Sep 2023 19:41:33 GMT

Hi everyone,

Looks like the push must have worked. We are now seeing the update sooner. See below:

+++++ SNIP +++++

A new comment was created on your recent case ( 02618638 ). To view the details of this case, provide input or add attachments, please click here.

Comment: Hi Raul,Greetings!I hope you are doing good.Thank you for your patience. The fix has been now implemented in below next releases below.10.2.6====> 9/28/2311.0.3====> 10/19/23Meanwhile, If you have any queries feel free to reach out to me, I will be glad to assist you.Have a nice day!

++++ END SNIP +++++++

Re: After Upgrading our PA-820 to 11.0.2, we're seeing lots of data on dns-base application.

AlbertHernandez — Mon, 25 Sep 2023 20:19:40 GMT

I'm seeing this as well on versions 10.2.4-h3, anyone else on this version that can verify? I saw a pattern and it seems to be every Monday around 6am PST to 10am PST

Re: After Upgrading our PA-820 to 11.0.2, we're seeing lots of data on dns-base application.

RaulTrujillo — Mon, 25 Sep 2023 20:58:19 GMT

I couldn't pinpoint a pattern. What I did find out is it began when our internal users began to use the network/Internet. The good thing is that a fix is on its way.