Hello all, I have a virtual machine Linux server version 6 (2.6.32-573.18.1.el6.x86_64)
We recently rebooted our servers during our maintenance window and after the reboot one server was not able to sync with the ntp server.
The server was running with +3 minutes ahead of the ntp server, we restarted ntpd service and ran ntpdate in between recycling ntpd.
That solved the problem, but what I am trying to find why this server did not sync with the ntp server.
We also found that the ESXI where the server was running had the wrong time (+3 minutes), and only 1 server out of the 20 running on the same esxi was not able to sync with the ntp server.

I tried to replicate the issue using a test server but in all the attempts to replicate the problem fails, the time gets adjusted to the ntp server with in a minute.

Here are some entries I grep out of the message log file, if any additional information is needed please let me know.

Any suggestions or theories as to what could have been the problem will be appreciated.

(server was rebooted twice you will see ntp restarted twice)

grep ntpd messages*

messages-20160918:Sep 17 12:11:42 rsagovprd1 ntpd[2762]: ntpd exiting on signal 15
messages-20160918:Sep 17 12:15:51 rsagovprd1 ntpd[3139]: ntpd 4.2.6p5@1.2349-o Tue May 3 15:12:50 UTC 2016 (1)
messages-20160918:Sep 17 12:15:51 rsagovprd1 ntpd[3140]: proto: precision = 0.059 usec
messages-20160918:Sep 17 12:15:51 rsagovprd1 ntpd[3140]: 0.0.0.0 c01d 0d kern kernel time sync enabled
messages-20160918:Sep 17 12:15:53 rsagovprd1 ntpd[3140]: Listen and drop on 0 v4wildcard 0.0.0.0 UDP 123
messages-20160918:Sep 17 12:15:53 rsagovprd1 ntpd[3140]: Listen and drop on 1 v6wildcard :: UDP 123
messages-20160918:Sep 17 12:15:53 rsagovprd1 ntpd[3140]: Listen normally on 2 lo 127.0.0.1 UDP 123
messages-20160918:Sep 17 12:15:53 rsagovprd1 ntpd[3140]: Listen normally on 3 eth0 172.16.95.193 UDP 123
messages-20160918:Sep 17 12:15:53 rsagovprd1 ntpd[3140]: Listen normally on 4 eth1 172.16.51.25 UDP 123
messages-20160918:Sep 17 12:15:53 rsagovprd1 ntpd[3140]: Listen normally on 5 lo ::1 UDP 123
messages-20160918:Sep 17 12:15:53 rsagovprd1 ntpd[3140]: Listen normally on 6 eth1 fe80::250:56ff:fe87:1285 UDP 123
messages-20160918:Sep 17 12:15:53 rsagovprd1 ntpd[3140]: Listen normally on 7 eth0 fe80::250:56ff:fe87:575 UDP 123
messages-20160918:Sep 17 12:15:53 rsagovprd1 ntpd[3140]: Listening on routing socket on fd #24 for interface updates
messages-20160918:Sep 17 12:15:55 rsagovprd1 ntpd[3140]: 0.0.0.0 c016 06 restart
messages-20160918:Sep 17 12:15:55 rsagovprd1 ntpd[3140]: 0.0.0.0 c012 02 freq_set ntpd -2.713 PPM
messages-20160918:Sep 17 12:15:55 rsagovprd1 ntpd[3140]: 0.0.0.0 c515 05 clock_sync
messages-20160918:Sep 17 12:23:48 rsagovprd1 ntpd[3140]: frequency error -968 PPM exceeds tolerance 500 PPM
messages-20160918:Sep 17 12:24:55 rsagovprd1 ntpd[3140]: frequency error -636 PPM exceeds tolerance 500 PPM
messages-20160918:Sep 17 12:26:04 rsagovprd1 ntpd[3140]: frequency error -640 PPM exceeds tolerance 500 PPM
messages-20160918:Sep 17 12:35:01 rsagovprd1 ntpd[3140]: frequency error -1585 PPM exceeds tolerance 500 PPM
messages-20160918:Sep 17 12:36:10 rsagovprd1 ntpd[3140]: frequency error -535 PPM exceeds tolerance 500 PPM
messages-20160918:Sep 17 14:12:19 rsagovprd1 ntpd[3140]: frequency error -624 PPM exceeds tolerance 500 PPM
messages-20160918:Sep 17 14:14:33 rsagovprd1 ntpd[3140]: frequency error -746 PPM exceeds tolerance 500 PPM
messages-20160918:Sep 17 14:16:41 rsagovprd1 ntpd[3140]: frequency error -734 PPM exceeds tolerance 500 PPM
messages-20160918:Sep 17 14:25:30 rsagovprd1 ntpd[3140]: frequency error -1459 PPM exceeds tolerance 500 PPM
messages-20160918:Sep 17 15:05:15 rsagovprd1 ntpd[3284]: ntpd 4.2.6p5@1.2349-o Tue May 3 15:12:50 UTC 2016 (1)
messages-20160918:Sep 17 15:05:15 rsagovprd1 ntpd[3285]: proto: precision = 0.123 usec
messages-20160918:Sep 17 15:05:15 rsagovprd1 ntpd[3285]: 0.0.0.0 c01d 0d kern kernel time sync enabled
messages-20160918:Sep 17 15:05:15 rsagovprd1 ntpd[3285]: Listen and drop on 0 v4wildcard 0.0.0.0 UDP 123
messages-20160918:Sep 17 15:05:15 rsagovprd1 ntpd[3285]: Listen and drop on 1 v6wildcard :: UDP 123
messages-20160918:Sep 17 15:05:15 rsagovprd1 ntpd[3285]: Listen normally on 2 lo 127.0.0.1 UDP 123
messages-20160918:Sep 17 15:05:15 rsagovprd1 ntpd[3285]: Listen normally on 3 eth0 172.16.95.193 UDP 123
messages-20160918:Sep 17 15:05:15 rsagovprd1 ntpd[3285]: Listen normally on 4 eth1 172.16.51.25 UDP 123
messages-20160918:Sep 17 15:05:15 rsagovprd1 ntpd[3285]: Listen normally on 5 lo ::1 UDP 123
messages-20160918:Sep 17 15:05:15 rsagovprd1 ntpd[3285]: Listen normally on 6 eth1 fe80::250:56ff:fe87:1285 UDP 123
messages-20160918:Sep 17 15:05:15 rsagovprd1 ntpd[3285]: Listen normally on 7 eth0 fe80::250:56ff:fe87:575 UDP 123
messages-20160918:Sep 17 15:05:15 rsagovprd1 ntpd[3285]: Listening on routing socket on fd #24 for interface updates
messages-20160918:Sep 17 15:05:15 rsagovprd1 ntpd[3285]: 0.0.0.0 c016 06 restart
messages-20160918:Sep 17 15:05:15 rsagovprd1 ntpd[3285]: 0.0.0.0 c012 02 freq_set ntpd -500.000 PPM
messages-20160918:Sep 17 15:05:16 rsagovprd1 ntpd[3285]: 0.0.0.0 c515 05 clock_sync
messages-20160918:Sep 17 15:05:18 rsagovprd1 ntpd[3285]: frequency error -554 PPM exceeds tolerance 500 PPM
messages-20160918:Sep 17 15:05:27 rsagovprd1 ntpd[3285]: frequency error -745 PPM exceeds tolerance 500 PPM
messages-20160918:Sep 17 15:10:56 rsagovprd1 ntpd[3285]: frequency error -9437 PPM exceeds tolerance 500 PPM
messages-20160918:Sep 17 15:19:54 rsagovprd1 ntpd[3285]: frequency error -15113 PPM exceeds tolerance 500 PPM
messages-20160918:Sep 17 15:21:02 rsagovprd1 ntpd[3285]: frequency error -2346 PPM exceeds tolerance 500 PPM
messages-20160918:Sep 17 15:35:30 rsagovprd1 ntpd[3285]: frequency error -6388 PPM exceeds tolerance 500 PPM
messages-20160918:Sep 17 15:37:41 rsagovprd1 ntpd[3285]: frequency error -1388 PPM exceeds tolerance 500 PPM
messages-20160918:Sep 17 15:53:01 rsagovprd1 ntpd[3285]: frequency error -6746 PPM exceeds tolerance 500 PPM
messages-20160918:Sep 17 16:10:26 rsagovprd1 ntpd[3285]: frequency error -7588 PPM exceeds tolerance 500 PPM
messages-20160918:Sep 17 16:17:02 rsagovprd1 ntpd[3285]: frequency error -11239 PPM exceeds tolerance 500 PPM
messages-20160918:Sep 17 16:21:35 rsagovprd1 ntpd[3285]: frequency error -7603 PPM exceeds tolerance 500 PPM
messages-20160918:Sep 17 16:25:59 rsagovprd1 ntpd[3285]: frequency error -7670 PPM exceeds tolerance 500 PPM
messages-20160918:Sep 17 16:31:27 rsagovprd1 ntpd[3285]: frequency error -9406 PPM exceeds tolerance 500 PPM
messages-20160918:Sep 17 16:37:59 rsagovprd1 ntpd[3285]: frequency error -11139 PPM exceeds tolerance 500 PPM
....
....
....
messages-20160925:Sep 21 07:06:48 rsagovprd1 ntpd[3285]: frequency error -9832 PPM exceeds tolerance 500 PPM
messages-20160925:Sep 21 07:10:03 rsagovprd1 ntpd[3285]: frequency error -3991 PPM exceeds tolerance 500 PPM
messages-20160925:Sep 21 07:12:55 rsagovprd1 ntpd[3285]: 0.0.0.0 0618 08 no_sys_peer
messages-20160925:Sep 21 07:18:47 rsagovprd1 ntpd[3285]: frequency error -6798 PPM exceeds tolerance 500 PPM
messages-20160925:Sep 21 07:25:19 rsagovprd1 ntpd[3285]: frequency error -7511 PPM exceeds tolerance 500 PPM
messages-20160925:Sep 21 07:29:46 rsagovprd1 ntpd[3285]: frequency error -5272 PPM exceeds tolerance 500 PPM
....
....
....
messages-20160925:Sep 21 17:58:04 rsagovprd1 ntpd[3285]: frequency error -1572 PPM exceeds tolerance 500 PPM
messages-20160925:Sep 21 18:00:23 rsagovprd1 ntpd[3285]: 0.0.0.0 0628 08 no_sys_peer
messages-20160925:Sep 21 18:03:31 rsagovprd1 ntpd[3285]: frequency error -3647 PPM exceeds tolerance 500 PPM
messages-20160925:Sep 21 18:04:35 rsagovprd1 ntpd[3285]: frequency error -1571 PPM exceeds tolerance 500 PPM
messages-20160925:Sep 21 18:13:20 rsagovprd1 ntpd[3285]: frequency error -9285 PPM exceeds tolerance 500 PPM