DNS Proxy Errors

[mario11584]

We have a remote office using a PA-200 in the middle east. I configured it to use DNS proxy with caching to lower the time for resolution over the VPN tunnel back to our corporate DNS servers in the US. We also have intermittent disconnects due to the unreliable internet connection there and this seemed to help eliminate some of the complaints of network connectivity problems. At any rate, I am receiving possibly thousands of errors in the system logs related to DNS proxy. Here is just 3 lines of it:

Here is a screenshot of my config. I also have a bunch of static entries under that tab and nothing under proxy rules.

It seems that things are resolving fine, however. From a Windows 8 VM, configured to use the DNS proxy only doesn't seem to be having any problems. Any thoughts?

[kadak]

Hello Mario,

Could you please attach the output for the following command in a notepad file:

> tail lines 1000 mp-log dnsproxyd.log

> debug dnsproxyd show connections

Hopefully, dnsproxyd.log gives us some valuable information about those failed resolutions.

Regards,

Kunal Adak.

[mario11584]

Tail lines shows the following around 3:10 on 9/24:

Sep 24 02:57:21 Error: pan_dnsproxyd_recv_server_udp_cb(pan_dnsproxy_udp.c:487): [Drop Rcvd Server Pkt]: Error in processing packet

Sep 24 04:21:52 Error: pan_dnsproxy_process_server_pkt(pan_dnsproxy_pkt.c:1320): [9951/-][Drop Rcvd Server Pkt]: No pending entry in conn tbl for server_tid:9951

Sep 24 04:21:52 Error: remove_conn_tbl_entry(pan_dnsproxy_pkt.c:284): conn_tbl[9951] entry is already freed!

Sep 24 04:21:52 Error: pan_dnsproxyd_recv_server_udp_cb(pan_dnsproxy_udp.c:487): [Drop Rcvd Server Pkt]: Error in processing packet

Sep 24 04:21:54 Error: pan_dnsproxy_process_server_pkt(pan_dnsproxy_pkt.c:1320): [5461/-][Drop Rcvd Server Pkt]: No pending entry in conn tbl for server_tid:5461

Sep 24 04:21:54 Error: remove_conn_tbl_entry(pan_dnsproxy_pkt.c:284): conn_tbl[5461] entry is already freed!

Sep 24 04:21:54 Error: pan_dnsproxyd_recv_server_udp_cb(pan_dnsproxy_udp.c:487): [Drop Rcvd Server Pkt]: Error in processing packet

Sep 24 04:38:40 Error: pan_dnsproxy_process_server_pkt(pan_dnsproxy_pkt.c:1320): [20840/-][Drop Rcvd Server Pkt]: No pending entry in conn tbl for server_tid:20840

Sep 24 04:38:40 Error: remove_conn_tbl_entry(pan_dnsproxy_pkt.c:284): conn_tbl[20840] entry is already freed!

Sep 24 04:38:40 Error: pan_dnsproxyd_recv_server_udp_cb(pan_dnsproxy_udp.c:487): [Drop Rcvd Server Pkt]: Error in processing packet

debug shows "no pending connections". I tried to initiate connections but I received the same results.

[kadak]

Hello Mario,

Thank you for providing the details regarding dnsproxyd.

This issue could be related to bursty DNS response received from the server, which would clog the buffer space available for DNS. This calls for a live troubleshooting session and in-depth tech support analysis - to see if a high rate would cause buffer depletion leading to dropped packets from the server side.  I was able to look up couple of similar existing cases which are still being investigated.

At this point, opening a case through support portal would be the best way to tackle your issue.

Regards,

Kunal Adak

[shasnain]

Hi Mario,

If the requests are very high, using alternative DNS like BIND can be a good option here.

Thanks,

Syed R Hasnain

[nayubi]

The above errors are due to a delayed response from the DNS server.  There is an error processing the response packet from the dns server because the entry has already been cleared out to the tables.  Try to use a server that has a faster response time to clear this up. 

Sep 24 04:21:54 Error: pan_dnsproxyd_recv_server_udp_cb(pan_dnsproxy_udp.c:487): [Drop Rcvd Server Pkt]: Error in processing packet

Sep 24 04:38:40 Error: pan_dnsproxy_process_server_pkt(pan_dnsproxy_pkt.c:1320): [20840/-][Drop Rcvd Server Pkt]: No pending entry in conn tbl for server_tid:20840

Sep 24 04:38:40 Error: remove_conn_tbl_entry(pan_dnsproxy_pkt.c:284): conn_tbl[20840] entry is already freed!

Sep 24 04:38:40 Error: pan_dnsproxyd_recv_server_udp_cb(pan_dnsproxy_udp.c:487): [Drop Rcvd Server Pkt]: Error in processing packet

If this was due to bursty traffic and the buffers were becoming depleted you would most likely get the following error:  Error: sendfromto(pan_dnsproxy_util.c:378): sendmsg (No buffer space available) 

[mario11584]

Any ideas on how to resolve the issue?

Palo Alto support is suggesting some type of vulnerability and traffic is being cut off. I don't see anything, at all, in the threat logs. It's suggested I remove the vulnerability profile from the security policy DNS traffic is using but if the threat logs don't show anything it doesn't seem like that would do the trick. Plus, I would be opening my network up to vulnerabilities. I would create an exception before completely removing a vulnerability profile.

[nayubi]

The logs indicate the server is slow to respond to the requests and they are being aged out.  This can only be fixed by response times, weather hardware upgrade, or adding additional servers, etc.  If you have multiple servers you may try and load balance between them by domains to lighten the load.  You can also enable caching on the advanced tab.  Around how many requests are you trying to proxy for?

Your server should be responding back to the pan dns requests via the management unless configured with a service route.  What is the vulnerability that this traffic is being seen as and on what interface and zone and direction is it seen coming from?  Is it the server or client traffic being identified as the threat.