FQDN Address Object wont resolve

[PhilH]

Hello,

I am trying to setup a U turn NAT that runs so that any system trying to contact time.apple.com using the NTP protocol will be rerouted to an internal NTP server. We do not allow NTP out and iPhones and iPads ignore DHCP settings for the NTP server.

I have created the NAT rule and when I input the destination as an IP address (not an address object) it works fine. When I use the defined address object time.apple.com it does not work.

I’ve logged into the CLI of the machine and run "request system fqdn show" I get

VSYS : vsys1 (using mgmt-obj dnsproxy object)
time.apple.com (Objectname time.apple.com):
Not resolved

This remains the same even if I run a manual refresh

Task list shows “Refresh FQDN Failed” with no further information. I can see on the internal DNS server that when the refresh runs it successfully executes the query to the DNS server.

If I ping time.apple.com from the CLI it immediately resolves and IP address.

DNS is setup on the management interface (Device - Setup – Services) pointing to an internal DNS server. Nslookup from an internal machine returns multiple IP addresses, so that DNS server is capable of resolution

There is no dnsproxy setup on the device.

It was suggested I allow the management IP to ping out (as it is blocked by security rules atm) however that doesn’t help.

time.apple.com does not feature in any security rules, only as an address object and in the NAT rule.

Searching google and the Palo Alto support site shows results but either the resolution is the same as I already have setup, or the errors aren’t for “Not resolved”. The rule is not being shadowed. There are no errors pertaining to this NAT rule when committing.

Ive run out of ideas, any help would be appreciated. I am running PAN OS 7.0.4

[santonic]

I have 'not resolved' for FQDN objects which aren't used in security policy. I'd say if they are in NAT policy they should be resolved. But just for fun try using the object somewhere in security policy too and see if it helps.

[reaper]

To try and figure out why fqdn would not refresh, please try setting the management-service logging to debug fqdn and tail the log while requesting a fqdn refresh:

> debug management-server on debug
> debug management-server set fqdn all
> tail follow yes mp-log ms.log

in a second CLI window then request the refresh:

> request system fqdn refresh force

Alternatively, if you have an internal DNS server you could poison the DNS record for time.apple.com to point to any IP you like or use a proxy-dns on the firewall to achieve that same goal

[PhilH]

Santonic, thankyou for the suggestion. I have thought the same as you with regards to the rule. I will keep your idea in mind if I dont get anywhere with Reapers suggestion.

Reaper, here is the output.

------------PAN DNSCFG FQDNS TO REFRESH-----------
----------- vsys1 (mgmt-obj)------------
                fqdn = 'time.apple.com'
------------FQDNS END-----------
 09:40:13.317 debug: pan_dnscfg_resolve_now(pan_cfg_dnscfg.c:3239): dnscfgmod: sending fqdns in vsys vsys1 to resolve using mgmt-obj
 09:40:13.317 debug: pan_dnscfg_resolve_now(pan_cfg_dnscfg.c:3300): dnscfgmod: Sending batch request # 1 for 1 fqdns
 09:40:13.317 debug: pan_dnsproxyd_sysd_client_query_send(pan_dnsproxyd_sysd_api.c:105): Sent DNS Proxy fqdn requests to daemon
 09:40:13.317 dnscfgmod:Fqdn refresh job 11891 scheduled
 09:40:13.367 debug: pan_dnscfg_resolve_now(pan_cfg_dnscfg.c:3335): dnscfgmod: Starting timedwait
 09:40:14.112 debug: pan_dnsproxyd_sysd_client_recv_cb(pan_dnsproxyd_sysd_api.c:538): change: notify obj 'sw.dnsproxyd.runtime.fqdn-resp'
 09:40:14.112 debug: pan_dnsproxyd_sysd_client_recv_cb(pan_dnsproxyd_sysd_api.c:553): Client recv cb: changed object received
 09:40:14.112 debug: pan_dnsproxyd_sysd_client_resp_recv(pan_dnsproxyd_sysd_api.c:418): Parse the sysd fqdn response received
 09:40:14.112 debug: pan_dnsproxyd_sysd_client_resp_recv(pan_dnsproxyd_sysd_api.c:429): Got obj-name:mgmt-obj
 09:40:14.112 debug: pan_dnsproxyd_sysd_client_resp_recv(pan_dnsproxyd_sysd_api.c:438): Got batch-num:1
 09:40:14.112 debug: pan_dnsproxyd_sysd_client_resp_recv(pan_dnsproxyd_sysd_api.c:443): Got num-fqdns:1
 09:40:14.112 debug: pan_dnsproxyd_sysd_client_resp_recv(pan_dnsproxyd_sysd_api.c:448): Got num-resolved:1
 09:40:14.112 debug: pan_dnsproxyd_sysd_client_resp_recv(pan_dnsproxyd_sysd_api.c:453): Got num-failed:0
 09:40:14.112 debug: pan_dnsproxyd_sysd_client_resp_recv(pan_dnsproxyd_sysd_api.c:465): Got fqdn:time.apple.com
 09:40:14.112 debug: pan_dnsproxyd_sysd_client_resp_recv(pan_dnsproxyd_sysd_api.c:476): Got ttl:606
 09:40:14.112 debug: pan_dnsproxyd_sysd_client_resp_recv(pan_dnsproxyd_sysd_api.c:481): Got ip_count:14
 09:40:14.112 debug: pan_dnsproxyd_sysd_client_resp_recv(pan_dnsproxyd_sysd_api.c:486): Got tstamp:0
 09:40:14.112 debug: pan_dnsproxyd_sysd_client_resp_recv(pan_dnsproxyd_sysd_api.c:504): Got ip: 17.253.38.253
 09:40:14.112 debug: pan_dnsproxyd_sysd_client_resp_recv(pan_dnsproxyd_sysd_api.c:504): Got ip: 17.253.34.253
 09:40:14.112 debug: pan_dnsproxyd_sysd_client_resp_recv(pan_dnsproxyd_sysd_api.c:504): Got ip: 17.253.68.253
 09:40:14.112 debug: pan_dnsproxyd_sysd_client_resp_recv(pan_dnsproxyd_sysd_api.c:504): Got ip: 17.253.52.253
 09:40:14.113 debug: pan_dnsproxyd_sysd_client_resp_recv(pan_dnsproxyd_sysd_api.c:504): Got ip: 17.253.14.253
 09:40:14.113 debug: pan_dnsproxyd_sysd_client_resp_recv(pan_dnsproxyd_sysd_api.c:504): Got ip: 17.253.24.253
 09:40:14.113 debug: pan_dnsproxyd_sysd_client_resp_recv(pan_dnsproxyd_sysd_api.c:504): Got ip: 17.253.54.251
 09:40:14.113 debug: pan_dnsproxyd_sysd_client_resp_recv(pan_dnsproxyd_sysd_api.c:504): Got ip: 17.253.84.253
 09:40:14.113 debug: pan_dnsproxyd_sysd_client_resp_recv(pan_dnsproxyd_sysd_api.c:504): Got ip: 17.253.12.253
 09:40:14.113 debug: pan_dnsproxyd_sysd_client_resp_recv(pan_dnsproxyd_sysd_api.c:504): Got ip: 17.253.2.253
 09:40:14.113 debug: pan_dnsproxyd_sysd_client_resp_recv(pan_dnsproxyd_sysd_api.c:504): Got ip: 17.253.26.253
 09:40:14.113 debug: pan_dnsproxyd_sysd_client_resp_recv(pan_dnsproxyd_sysd_api.c:504): Got ip: 17.253.6.253
 09:40:14.113 debug: pan_dnsproxyd_sysd_client_resp_recv(pan_dnsproxyd_sysd_api.c:504): Got ip: 17.253.4.253
 09:40:14.113 debug: pan_dnsproxyd_sysd_client_resp_recv(pan_dnsproxyd_sysd_api.c:504): Got ip: 17.253.20.253
 09:40:14.113 debug: pan_dnsproxyd_sysd_client_resp_recv(pan_dnsproxyd_sysd_api.c:523): Parsed sysd dnsproxy fqdn response
 09:40:14.113 debug: pan_dnsproxyd_sysd_client_recv_cb(pan_dnsproxyd_sysd_api.c:577): Client recv cb: calling client callback
 09:40:14.113 debug: pan_dnscfg_recv_resp(pan_cfg_dnscfg.c:380): dnscfgmod: pan_dnscfg_recv_resp: batch 1, mgmt-obj, vec size 1, actual size 1, resolved 1, failed 0
 09:40:14.113 debug: pan_dnscfg_recv_resp(pan_cfg_dnscfg.c:406): dnscfgmod: ----------------Response received for batch 1, resolved 1, failed 0 -------------
 09:40:14.113 debug: pan_dnscfg_recv_resp(pan_cfg_dnscfg.c:430): dnscfgmod:     Fqdn time.apple.com 17.253.38.253 TTL = 606, 17.253.34.253 TTL = 606, 17.253.68.253 TTL = 606, 17.253.52.253 TTL = 606, 17.253.14.253 TTL = 606, 17.253.24.253 TTL = 606, 17.253.54.251 TTL = 606, 17.253.84.253 TTL = 606, 17.253.12.253 TTL = 606, 17.253.2.253 TTL = 606, 17.253.26.253 TTL = 606, 17.253.6.253 TTL = 606, 17.253.4.253 TTL = 606, 17.253.20.253 TTL = 606,
 09:40:14.113 debug: pan_dnscfg_recv_resp(pan_cfg_dnscfg.c:462): dnscfgmod: Looking for fqdn time.apple.com in target
 09:40:14.113 debug: pan_dnscfg_recv_resp(pan_cfg_dnscfg.c:473): dnscfgmod: Fqdn time.apple.com is now resolved !
 09:40:14.113 debug: pan_dnscfg_recv_resp(pan_cfg_dnscfg.c:514): dnscfgmod: Adding 17.253.38.253 to resolved ips list for time.apple.com/time.apple.com
 09:40:14.113 debug: pan_dnscfg_recv_resp(pan_cfg_dnscfg.c:514): dnscfgmod: Adding 17.253.34.253 to resolved ips list for time.apple.com/time.apple.com
 09:40:14.113 debug: pan_dnscfg_recv_resp(pan_cfg_dnscfg.c:514): dnscfgmod: Adding 17.253.68.253 to resolved ips list for time.apple.com/time.apple.com
 09:40:14.113 debug: pan_dnscfg_recv_resp(pan_cfg_dnscfg.c:514): dnscfgmod: Adding 17.253.52.253 to resolved ips list for time.apple.com/time.apple.com
 09:40:14.113 debug: pan_dnscfg_recv_resp(pan_cfg_dnscfg.c:514): dnscfgmod: Adding 17.253.14.253 to resolved ips list for time.apple.com/time.apple.com
 09:40:14.113 debug: pan_dnscfg_recv_resp(pan_cfg_dnscfg.c:514): dnscfgmod: Adding 17.253.24.253 to resolved ips list for time.apple.com/time.apple.com
 09:40:14.113 debug: pan_dnscfg_recv_resp(pan_cfg_dnscfg.c:514): dnscfgmod: Adding 17.253.54.251 to resolved ips list for time.apple.com/time.apple.com
 09:40:14.113 debug: pan_dnscfg_recv_resp(pan_cfg_dnscfg.c:514): dnscfgmod: Adding 17.253.84.253 to resolved ips list for time.apple.com/time.apple.com
 09:40:14.113 debug: pan_dnscfg_recv_resp(pan_cfg_dnscfg.c:514): dnscfgmod: Adding 17.253.12.253 to resolved ips list for time.apple.com/time.apple.com
 09:40:14.113 debug: pan_dnscfg_recv_resp(pan_cfg_dnscfg.c:514): dnscfgmod: Adding 17.253.2.253 to resolved ips list for time.apple.com/time.apple.com
 09:40:14.113 debug: pan_dnscfg_recv_resp(pan_cfg_dnscfg.c:514): dnscfgmod: Adding 17.253.26.253 to resolved ips list for time.apple.com/time.apple.com
 09:40:14.113 debug: pan_dnscfg_recv_resp(pan_cfg_dnscfg.c:514): dnscfgmod: Adding 17.253.6.253 to resolved ips list for time.apple.com/time.apple.com
 09:40:14.113 debug: pan_dnscfg_recv_resp(pan_cfg_dnscfg.c:514): dnscfgmod: Adding 17.253.4.253 to resolved ips list for time.apple.com/time.apple.com
 09:40:14.113 debug: pan_dnscfg_recv_resp(pan_cfg_dnscfg.c:514): dnscfgmod: Adding 17.253.20.253 to resolved ips list for time.apple.com/time.apple.com
 09:40:14.113 debug: pan_dnscfg_recv_resp(pan_cfg_dnscfg.c:529): dnscfgmod: response received 1, to resolve 1
 09:40:14.113 debug: pan_dnscfg_recv_resp(pan_cfg_dnscfg.c:531): dnscfgmod: All Fqdns responses received
 09:40:14.113 debug: pan_dnscfg_resolve_now(pan_cfg_dnscfg.c:3341): dnscfgmod: Done timedwait
 09:40:14.113 dnscfgmod: Resolving fqdns took 1 secs
 09:40:14.113 Fqdn refresher thread device requested last config
 09:40:14.435 debug: pan_dnscfg_replace_addresses(pan_cfg_dnscfg.c:2249): dnscfgmod: pan_dnscfg_replace_addresses: Replacing FQDNs in vsys1
 09:40:14.436 debug: pan_dnscfg_convert_fqdns(pan_cfg_dnscfg.c:2728): dnscfgmod: replaced fqdns unders all vsyses
 09:40:14.436 debug: pan_dnscfg_replace_addresses(pan_cfg_dnscfg.c:2249): dnscfgmod: pan_dnscfg_replace_addresses: Replacing FQDNs in shared
 09:40:14.436 debug: pan_dnscfg_replace_addresses(pan_cfg_dnscfg.c:2262): dnscfgmod: Failed to get addressnodes in shared
 09:40:14.436 debug: pan_dnscfg_convert_fqdns(pan_cfg_dnscfg.c:2738): dnscfgmod: replaced fqdns unders shared
 09:40:14.606 debug: pan_cfg_refresh_deviceconfig(pan_cfg_commit_jobs.c:2934): Refresh send to device config
 09:40:14.918 debug: pan_cfg_refresh_deviceconfig(pan_cfg_commit_jobs.c:2984): deviceconfig string to xml takes 0 seconds to complete
 09:40:16.081 debug: _pan_mgmt_client_send_phase1(pan_cfg_commit_jobs.c:1502): for client device get transformed config takes 2 seconds to complete
 09:40:16.082 debug: _pan_mgmt_client_send_phase1(pan_cfg_commit_jobs.c:1504): config for client device is 7674022 bytes long
 09:40:16.163 debug: _pan_mgmt_client_send_phase1(pan_cfg_commit_jobs.c:1528): for client device send config takes 0 seconds to complete
 09:40:17.992 debug: pan_comm_lcs_get_next_addr(cs_conn.c:4345):  >>> pan_comm_lcs_get_next_addr()
 09:40:19.635 client device reported error: <<vsys1>>
vsys1: Rule 'Salesforce' application dependency warning:
        Application 'salesforce-base' requires 'ssl' be allowed
        Application 'salesforce-chatter' requires 'ssl' be allowed
        Application 'salesforce-chatter' requires 'web-browsing' be allowed
vsys1: Rule 'Allowed Applications' application dependency warning:
        Application 'hotmail' requires 'silverlight' be allowed
        Application 'hotmail' requires 'ssl' be allowed
        Application 'hotmail' requires 'web-browsing' be allowed
        Application 'twitter-base' requires 'ssl' be allowed
        Application 'linkedin-intro' requires 'imap' be allowed
        Application 'linkedin-intro' requires 'smtp' be allowed
vsys1: Rule 'Lync' application dependency warning:
        Application 'ms-lync-base' requires 'ssl' be allowed
        Application 'ms-lync-audio' requires 'rtcp' be allowed
        Application 'ms-lync-audio' requires 'rtp-base' be allowed
        Application 'ms-lync-video' requires 'rtcp' be allowed
        Application 'ms-lync-video' requires 'rtp-base' be
 09:40:19.636 debug: pan_mgmt_client_err_callback(pan_cfg_commit_jobs.c:810): Finally received err msgs sent by devsrvr, notify main control
 09:40:19.637 client device reported Phase 1 FAILED
 09:40:19.637 debug: pan_cfg_refresh_deviceconfig(pan_cfg_commit_jobs.c:3020): Takes 5 seconds to complete Phase 1
 09:40:19.637 Error:  pan_cfg_refresh_deviceconfig(pan_cfg_commit_jobs.c:3031): phase 1 failed  cstate:6 -  verify:0
 09:40:19.637 debug: pan_cfg_refresh_deviceconfig(pan_cfg_commit_jobs.c:3057): Sent p1 abort to dev server.
 09:40:19.637 Error:  pan_dnscfg_force_refresh_fqdns_after_fail(pan_cfg_dnscfg.c:3753): Trying to refresh fqdn job after the first retry.Not allowed.
 09:40:19.639 client device reported error: Config commit phase 1 aborted(Module: device)
 09:40:19.639 Error:  pan_mgmt_client_err_callback(pan_cfg_commit_jobs.c:800): but there was no outstanding Phase 0/Phase 1/Phase 2. Ignoring - verify: 0
 09:40:19.693 Error:  pan_cfg_dnscfg_refresh_fqdns(pan_cfg_dnscfg.c:4338): Failed to refresh the fqdn.
 09:40:19.750 Error:  pan_jobmgr_process_job(pan_job_mgr.c:2279): Fqdn Refresh job failed
 09:40:19.751 debug: pan_jobmgr_thread(pan_job_mgr.c:2453): Consumer:list is empty, waiting for jobs
 09:40:37.993 debug: pan_comm_lcs_get_next_addr(cs_conn.c:4345):  >>> pan_comm_lcs_get_next_addr()

(I removed the date and timezone to make it easier to read)

That is confusing me even more. You can see it resolves the domain names and makes the entries but cant commit them? We always have dependency warnings on rules, but that shouldnt stop what we are trying to do, they are warnings not errors.

If you have any ideas I would appreciate them.

[santonic]

Similiar messages about phase 1 and commits failing were appearing on old devices with low memory (PA-2000, PA-500 1GB) sometimes. Restarting management plane helped with that. Do your other commits fail or succede?

Yeah, dependency warnins shouldn't be an issue here.

[reaper]

looks like the FQDN refresh itself is working as expected but the commit fails

Santonic's idea would be a good start as the dependency will not block the commit:

> debug software restart process management-server

and after a few minutes when the management has restarted

> request system fqdn refresh force yes

[PhilH]

Hi Santonic and Reaper,

All previous commits have succeeded. I have done what you suggested and restarted the management plane. Now when I type 

>request system fqdn refresh force yes

it returns

No FQDNs are used in rules, skipping refresh.

and when i type 

>request system fqdn show

it returns 

time.apple.com  (Objectname time.apple.com):

                        Not used

So it seems that I need to have it in a security rule for it to resolve the IPs, NAT rules don't count. What must have been happening before is that it would resolve the IPs and then during the commit realise that the FQDN isnt used in a security rule and registered the error.

Thankyou both for your help. It was much appreciated and I learnt how to do some more advanced troubleshooting in the process.

-Phil

[PhilH]

Further developments:

It seems that the error has something to do with my attempt at a NAT rule for this. My rule was to say "any NTP traffic going to time.apple.com to be redirected to an internal NTP server". This was working when i hard coded an IP for time.apple.com in there. I wanted to make this more robust so I tried changing this to the address object.

I created the security rule to allow time.apple.com to be contacted for NTP. It would not resolve until i disabled my NAT rule. I changed my NAT rule and sometimes i would get the commit error of

Mismatch of destination address translation range between original address and translated address

I found this explanation Here

It seems that the NAT rules do not like it when you set the Destination Address to be a FQDN Address object.

[santonic]

This error is indicating, that the number of addresses for source and destination you are translating are not the same and you are using static NAT. In general it has nothing to do with FQDN object. But I guess FQDN object has more than one IP and you are translating to only single address. And this can't be done as static NAT. And destination NAT can only be static unfortunatelly.

I guess you're stuck with several DNAT rules without FQDN objects and hoping IPs won't change much.

[PhilH]

Hi Santonic,

I agree. It would just be better if my misconfigured NAT threw a warning or error when committing instead of just breaking the DNS lookup.

For reference the NAT policy was 

From: Trust

To: Untrust

Destination Interface: ethernet1/1 (our external facing interface)

Destination Address: time.apple.com

Service: NTP

Source Translation: None

Destination Translation: Address <IP of internal NTP>

Thank you, both of you, for your help.

-Phil