- Access exclusive content
- Connect with peers
- Share your expertise
- Find support resources
09-15-2022 11:12 PM - edited 09-15-2022 11:19 PM
Hi Guys,
NTP was working well. But when authentication was enabled below msg is seen on the Firewall (NTP Stopped working)
NTP server is a local one using IP address (not FQDN)
PAN-OS Version 10.1.5-h1
All the other devices are syncing except for the Firewall.
Has anyone else seen this issue?
Any help would be greatly appreciated.
Regards,
11-06-2022 04:23 PM
@vzamy The issue has not been resolved.
Thanks for your suggestions they definitely helped us troubleshoot the issue. Initially, there was no 2-way communication but when the encryption was changed to SHA1 with exactly 40bit character length key the issue got resolved.
09-18-2022 12:53 AM
Hi @Pras ,
Based on the panos version you are running, I would first say upgrade to the latest recommended asap. (10.1.5 had it's issues)
If after upgrading the panos you are still experiencing the issue, please provide more info.
What type of troubleshooting have you done?
- Per your output, it shows that the NTP is not reachable, did you rule that out?
- What type of authentication are you using? md5/sha1?
- Did you try using "show counter global filter delta yes packet-filter yes" combined with a pcap filter. (mandatory)
- Do you have a pcap? what do you see there?
regards,
09-19-2022 06:25 PM - edited 09-19-2022 06:28 PM
Hi @vzamy,
Thanks for your reply. Please find my replies as below:
This deployment was working well before. But when customer tried to implement authentication, it stopped. Even now, this works well if the authentication is removed. (no upgrades were performed)
- Per your output, it shows that the NTP is not reachable, did you rule that out?
it is reachable- it's only not reachable when authentication is enabled
- What type of authentication are you using? md5/sha1?
Tried on both md5/sha- both not working
- Did you try using "show counter global filter delta yes packet-filter yes" combined with a pcap filter. (mandatory)
Not yet- since it works well with no authentication
- Do you have a pcap? what do you see there?
Not Yet (even if I take what should I be looking at in it?)
Thanks again.
09-21-2022 01:10 PM - edited 09-21-2022 01:30 PM
Hi @Pras
See my suggestions below:
- NTP authentication checks the authenticity of NTP server before synchronizing local time with the server, so I would double check the ntp auth type (md5/sha1), the symmetric auth-key (the shared password), and the NTP version.
- pcap wise, I would start with the very basic which is confirming I see a request/reply (although you mentioned it was working before w/o the auth). Next, I would inspect to see what NTP version is used and review the NTP header flags.
- Do you have service routes configured? Do you use the mgmt interface or other for NTP?
- If possible try switching to a different public ntp?
- Do you have two ntp server configured? if yes, either try removing one or make sure the key-id is different.
- filter under "monitor > logs > system" using (subtype eq ntpd) for any information regarding the ntp events.
- try "debug software restart process ntp" on the firewall and then try the show command again, if it still fails check the dagger log again.
regards,
09-29-2022 08:14 PM - edited 09-29-2022 08:17 PM
Hi @vzamy,
Had already done the restart of the ntp process with no luck.
The troubleshooted some more and would like to show come snaps as below:
NTP server is a Linux base and OS version is Redhat Enterprise Linux 7.9
It is pinging the server but apart from pinging nothing is being captured.
Also,
Server side has no issue apparently: it is getting the ntp packets from the FW
What could be the issue here?
Thanks in advance.
09-30-2022 01:31 AM - edited 09-30-2022 01:32 AM
Hi @Pras
- your printscreen shows one NTP call (the 1st line) so it does capture the NTP traffic.
Remember that the ntp polling occurs every x interval (between 64 and 1024)
- your tcpdump filter only captures destination x.x.x.134 thus we can't see any reponse from the otherside
e.g we can only see the ICMP requests
- please run the tcpdump like this instead
> tcpdump filter "host x.x.x.134" snaplen 0
- in the traffic logs, please extend one of the sessions using the detailed log view and inspect the bytes sent/received
what are they? it should show as 1:1 or higher, the goal is to see if we have bi-directional traffic.
- Please confirm if the Linux server is responsive (NTP-wise). The server-side printscreen you attached only shows a
unidirectional packet capture. Re-run the tcpdump on the server side:
e.g
tcpdump -nni 0.0 port 123
or
tcpdump -nni 0.0 host x.x.x.5
- Are the key-ids aligned between the FW and NTP server? To confirm, inspect the pcap and look for the Key ID sent from FW and
also, the Key ID returned from the server.
If they aren't try setting them both to the same key-id.
- The authentication procedures require that both the local and remote servers share the same key and key identifier for this purpose,
although different keys can be used with different servers. The key arguments are 32-bit unsigned integers with values from 1 to 65,534.
- since you have the MP attached to the DP you can run the PCAP there and download it for futher inspection. Make sure to configure
all the stages and observe if anything goes into the drop stage.
- If no NTP traffic is seen, to trigger it either restart the FW's ntp service or do a "commit force".
- Please also use "show counter global filter delta yes packet-filter yes" combined with a pcap filter. (mandatory)
and share the findings (make sure to generate the ntp traffic)
- If we still have no progress, at this stage I would try and rule out the NTP server. If you have a Cisco at hand
you can easily configure it to act as an NTP server.
10-10-2022 10:10 PM - edited 10-17-2022 03:01 PM
Hi @vzamy ,
Just received few requested outputs as below:
1.Restart done
admin@W01(active)> debug software restart process ntp
Process ntp was restarted by user admin
2.TCP dump (FW side)--> server I will share shortly
admin@W01(active)> tcpdump filter "host x.x.x.134" snaplen 0
Press Ctrl-C to stop capturing
tcpdump: listening on eth0, link-type EN10MB (Ethernet), capture size 65535 bytes
^C3 packets captured
3 packets received by filter
0 packets dropped by kernel
admin@W01(active)> view-pcap mgmt-pcap mgmt.pcap
14:39:21.753391 IP ...local.ntp > M01.x.local.ntp: NTPv4, Client, length 68
14:40:25.753435 IP ....local.ntp > M01x.local.ntp: NTPv4, Client, length 68
14:41:30.753456 IP ....local.ntp > RSSITDC1LM01.x.local.ntp: NTPv4, Client, length 68
admin@AW01(active)>
admin@W01(active)> show ntp
NTP state:
NTP not synched, using local clock
NTP server: x.x.x.134
status: rejected
reachable: no
authentication-type: symmetric key
Clock is set manually (i.e aligned to real time).
Server side captures show as below. I see that the NTP version configured from server side is 3. Could this be an issue?
Also, the reference ID is different (do not know if this could be an issue)
Thank you heaps in advance.
10-21-2022 12:12 PM - edited 10-21-2022 12:24 PM
Hi @Pras
- There seems to be NTP communication between the client (FW) and the server, your server-side captures confirm that.
- Reference ID (refid): 32-bit code identifying the particular server or reference clock. So this isn't the issue.
- keyid: Symmetric key ID for the 128-bit MD5 key used to generate and verify the MAC.
The client and server or peer can use different values, but they must map to the same key.
Please try resetting your PSK (use something simple for the testing purpose)
- Have a look at the current NTPv4 RFC 5905 “Network Time Protocol Version 4: Protocol and Algorithms Specification” in order to understand the packets and protocol details.
Looking at the wire you should understand the packet header (section 7.3 in the RFC).
- your "show ntp" output shows the server NTP is being 'rejected'.
admin@W01(active)> show NTP
NTP state:
NTP not synched, using local clock
NTP server: x.x.x.134
status: rejected << here
reachable: no
authentication-type: symmetric key
- NTP will refuse to synch if the time is too far off, due to its methodology, which is to slow or speed the clock and adjust gradually.
I see you mentioned you've manually adjusted your clock to a time closer to actual time for you time zone.
- NTPv4 is an extension of NTPv3 that supports IPv4 and IPv6. It is backward compatible with NTPv3, offers some new features, and time synchronization
is faster and more precise. Security has improved, NTPv4 supports public key cryptography and standard X509 certificates.
So although server side and client side use different NTP version this should not be an issue. But to rule out any possibility
I would align these two.
- Please comment on these two previously proposed steps:
1- Please also use "show counter global filter delta yes packet-filter yes" combined with a pcap filter. (mandatory)
and share the findings (make sure to generate the NTP traffic)
2- If we still have no progress, at this stage I would try and rule out the NTP server. If you have a Cisco at hand
you can easily configure it to act as an NTP server.
11-06-2022 04:23 PM
@vzamy The issue has not been resolved.
Thanks for your suggestions they definitely helped us troubleshoot the issue. Initially, there was no 2-way communication but when the encryption was changed to SHA1 with exactly 40bit character length key the issue got resolved.
06-18-2024 03:36 PM
How did you generate the sha1 ntp hash? I tried generating a hash from the plain text password via a linux command, and subsequently tried in powershell. Neither of those hashes worked. I know the plaintext password works when I apply it to other devices, but I can't seem to get the right hash for the sha1 password field.
Any help would be greatly appreciated!
Thanks,
Steve
Click Accept as Solution to acknowledge that the answer to your question has been provided.
The button appears next to the replies on topics you’ve started. The member who gave the solution and all future visitors to this topic will appreciate it!
These simple actions take just seconds of your time, but go a long way in showing appreciation for community members and the LIVEcommunity as a whole!
The LIVEcommunity thanks you for your participation!