NTP not working once authentication is enabled

[Pras]

Hi Guys,

NTP was working well. But when authentication was enabled below msg  is seen on the Firewall (NTP Stopped working)

NTP server is a local one using IP address (not FQDN)

PAN-OS Version 10.1.5-h1

All the other devices are syncing except for the Firewall.

Has anyone else seen this issue?

Any help would be greatly appreciated.

Regards,

@vzamy 

[vzamy]

Hi @Pras ,

Based on the panos version you are running, I would first say upgrade to the latest recommended asap. (10.1.5 had it's issues)

If after upgrading the panos you are still experiencing the issue, please provide more info.

What type of troubleshooting have you done?

- Per your output, it shows that the NTP is not reachable, did you rule that out?

- What type of authentication are you using? md5/sha1?

- Did you try using "show counter global filter delta yes packet-filter yes" combined with a pcap filter. (mandatory)

- Do you have a pcap? what do you see there?

regards,

[Pras]

Hi @vzamy,

Thanks for your reply. Please find my replies as below:

This deployment was working well before. But when customer tried to implement  authentication, it stopped. Even now, this works well if the authentication is removed. (no upgrades were performed)

- Per your output, it shows that the NTP is not reachable, did you rule that out? 

it is reachable- it's only not reachable when authentication is enabled

- What type of authentication are you using? md5/sha1?

Tried on both md5/sha- both not working

- Did you try using "show counter global filter delta yes packet-filter yes" combined with a pcap filter. (mandatory)

Not yet- since it works well with no authentication

- Do you have a pcap? what do you see there?

Not Yet (even if I take what should I be looking at in it?)

Thanks again.

[vzamy]

Hi  @Pras 

See my suggestions below:

- NTP authentication checks the authenticity of NTP server before synchronizing local time with the server, so I would double check the ntp auth type (md5/sha1), the symmetric auth-key (the shared password), and the NTP version.

- pcap wise, I would start with the very basic which is confirming I see a request/reply (although you mentioned it was working before w/o the auth). Next, I would inspect to see what NTP version is used and review the NTP header flags.

- Do you have service routes configured? Do you use the mgmt interface or other for NTP?

- If possible try switching to a different public ntp?

- Do you have two ntp server configured? if yes, either try removing one or make sure the key-id is different.

- filter under "monitor > logs > system" using (subtype eq ntpd) for any information regarding the ntp events.

- try "debug software restart process ntp" on the firewall and then try the show command again, if it still fails check the dagger log again.

regards,

[Pras]

Hi @vzamy,

Had already done the restart of the ntp process with no luck.

The troubleshooted some more and would like to show come snaps as below:

NTP server is a Linux base and OS version is Redhat Enterprise Linux 7.9

It is pinging the server but apart from pinging nothing is being captured.

Also,

Server side has no issue apparently: it is getting the ntp packets from the FW

What could be the issue here?

Thanks in advance.

[vzamy]

Hi @Pras 
- your printscreen shows one NTP call (the 1st line) so it does capture the NTP traffic.
Remember that the ntp polling occurs every x interval (between 64 and 1024)
- your tcpdump filter only captures destination x.x.x.134 thus we can't see any reponse from the otherside
e.g we can only see the ICMP requests
- please run the tcpdump like this instead
> tcpdump filter "host x.x.x.134" snaplen 0
- in the traffic logs, please extend one of the sessions using the detailed log view and inspect the bytes sent/received
what are they? it should show as 1:1 or higher, the goal is to see if we have bi-directional traffic.
- Please confirm if the Linux server is responsive (NTP-wise). The server-side printscreen you attached only shows a
unidirectional packet capture. Re-run the tcpdump on the server side:
e.g
tcpdump -nni 0.0 port 123
or
tcpdump -nni 0.0 host x.x.x.5

- Are the key-ids aligned between the FW and NTP server? To confirm, inspect the pcap and look for the Key ID sent from FW and
also, the Key ID returned from the server.
If they aren't try setting them both to the same key-id.
- The authentication procedures require that both the local and remote servers share the same key and key identifier for this purpose,
although different keys can be used with different servers. The key arguments are 32-bit unsigned integers with values from 1 to 65,534.
- since you have the MP attached to the DP you can run the PCAP there and download it for futher inspection. Make sure to configure
all the stages and observe if anything goes into the drop stage.
- If no NTP traffic is seen, to trigger it either restart the FW's ntp service or do a "commit force".
- Please also use "show counter global filter delta yes packet-filter yes" combined with a pcap filter. (mandatory)
and share the findings (make sure to generate the ntp traffic)
- If we still have no progress, at this stage I would try and rule out the NTP server. If you have a Cisco at hand
you can easily configure it to act as an NTP server.

[Pras]

Hi @vzamy ,

Just received few requested outputs as below:

1.Restart done

admin@W01(active)> debug software restart process ntp

Process ntp was restarted by user admin

2.TCP dump (FW side)--> server I will share shortly

admin@W01(active)> tcpdump filter "host x.x.x.134" snaplen 0

Press Ctrl-C to stop capturing 

tcpdump: listening on eth0, link-type EN10MB (Ethernet), capture size 65535 bytes

^C3 packets captured

3 packets received by filter

0 packets dropped by kernel

admin@W01(active)> view-pcap mgmt-pcap mgmt.pcap

14:39:21.753391 IP ...local.ntp > M01.x.local.ntp: NTPv4, Client, length 68

14:40:25.753435 IP ....local.ntp > M01x.local.ntp: NTPv4, Client, length 68

14:41:30.753456 IP ....local.ntp > RSSITDC1LM01.x.local.ntp: NTPv4, Client, length 68

admin@AW01(active)>

admin@W01(active)> show ntp

NTP state:

    NTP not synched, using local clock

    NTP server: x.x.x.134

        status: rejected

        reachable: no

        authentication-type: symmetric key

Clock is set manually (i.e aligned to real time).

Server side captures show as below. I see that the NTP version configured from server side is 3. Could this be an issue?

Also, the reference ID is different (do not know if this could be an issue)

Thank you heaps in advance.

[vzamy]

Hi @Pras 

- There seems to be NTP communication between the client (FW) and the server, your server-side captures confirm that.
- Reference ID (refid): 32-bit code identifying the particular server or reference clock. So this isn't the issue.
- keyid: Symmetric key ID for the 128-bit MD5 key used to generate and verify the MAC.
The client and server or peer can use different values, but they must map to the same key.
Please try resetting your PSK (use something simple for the testing purpose)
- Have a look at the current NTPv4 RFC 5905 “Network Time Protocol Version 4: Protocol and Algorithms Specification” in order to understand the packets and protocol details.
Looking at the wire you should understand the packet header (section 7.3 in the RFC).

- your "show ntp" output shows the server NTP is being 'rejected'.
admin@W01(active)> show NTP
NTP state:
NTP not synched, using local clock
NTP server: x.x.x.134
status: rejected << here
reachable: no
authentication-type: symmetric key

- NTP will refuse to synch if the time is too far off, due to its methodology, which is to slow or speed the clock and adjust gradually.
I see you mentioned you've manually adjusted your clock to a time closer to actual time for you time zone.

- NTPv4 is an extension of NTPv3 that supports IPv4 and IPv6. It is backward compatible with NTPv3, offers some new features, and time synchronization
is faster and more precise. Security has improved, NTPv4 supports public key cryptography and standard X509 certificates.
So although server side and client side use different NTP version this should not be an issue. But to rule out any possibility
I would align these two.

- Please comment on these two previously proposed steps:
1- Please also use "show counter global filter delta yes packet-filter yes" combined with a pcap filter. (mandatory)
and share the findings (make sure to generate the NTP traffic)
2- If we still have no progress, at this stage I would try and rule out the NTP server. If you have a Cisco at hand
you can easily configure it to act as an NTP server.

[Pras · Accepted solution]

@vzamy The issue has not been resolved.
Thanks for your suggestions they definitely helped us troubleshoot the issue. Initially, there was no 2-way communication but when the encryption was changed to SHA1  with  exactly 40bit character length key the issue got resolved. 

[cullums]

How did you generate the sha1 ntp hash? I tried generating a hash from the plain text password via a linux command, and subsequently tried in powershell. Neither of those hashes worked. I know the plaintext password works when I apply it to other devices, but I can't seem to get the right hash for the sha1 password field.

Any help would be greatly appreciated!

Thanks,

Steve

[bartpmika]

Do not hash the key value. Some NTP providers (eg. NIST) supply the authentication key value in ASCII format. The 40-character SHA1 NTP authentication key that you enter into the PanOS WebUI must be a Hex representation of the plaintext ASCII password, not its SHA1 hash. That 20 character ASCII password needs to be converted to a 40 character hex format. This site can help: https://www.rapidtables.com/convert/number/ascii-to-hex.html