How to clear the Leap Second Insertion flag after it has been received?

Updated -

Introduction

For the primary documentation on leap seconds, including a list of known issues, see Resolve Leap Second Issues in Red Hat Enterprise Linux.

While the RHEL kernel accepts a leapsecond for any month, ntpd in RHEL does not hand over the leapsecond in all cases. The ntp version ntp-4.2.4p8 sets the kernel leap bit only on end of June or December. The leap information will be forwarded to the clients on any day though. ntp Versions 4.2.6 and later set the kernel leap bit on end of any month, but require that a majority of its servers agree on the leap instead of only one as in 4.2.4 and before.

NTP servers can issue a leap second at the end of any month. You should check your ntp servers to see if they are aligned to issue a leap second:

[root@host ~]# ntpq -c "lassoc" -c "mrv &1 &999 leap,srcadr,stratum"

When you see leap=01, you have a leap second coming your way. Probably good idea to start checking at the end of every month and currently imperative to check at the end of the months of June and December.

To verify if the local system has currently leapsecond bits set:

[root@host ~]# ntptime |egrep 'INS|DEL'
ntp_gettime() returns code 1 (INS)
ntp_adjtime() returns code 1 (INS)
  status 0x2011 (PLL,INS,NANO),

Preventing Issues and Workarounds

First, see if you need to update Leap Second Issue Detector Red Hat Labs Tool

Red Hat strongly recommends that customers update their systems to the most recent available Red Hat Enterprise kernels. That said, however, the following two NTP client workarounds may be used with any systems that require precise, but not accurate, timestamp information. (In this context, "precise" means "a measure of the detail in which a quantity is expressed." "Accurate" means the "measurement deviation from true value.")

  • Workaround 1:

    • stop ntpd
    • run ntptime -s 0 to reset the kernel status
    • run ntptime and verify it doesn't print INS or DEL on the status line
    • run the system now without ntp active during the leap second
    • at some point in the future you might want to sync the systems idea of time again with the rest of the world. These methods could be used:

      • stepping the clock: when running ntpd again, the clock will be stepped
      • correct the clock slowly: ntpdate -B <some-ntp-server> could be used, the correction takes about 2000 seconds. After that, ntpd can be started again and the error will be already corrected, so ntpd won't need to perform any steps.
      • correct the clock faster: This uses the adjtimex tick value. The following script slows down the clock by 9% for 10.1 seconds. It can be used instead of ntpdate in the workaround I described earlier:

        #!/bin/bash
        tick=$(tickadj | awk '{print $3}')
        echo initial tick $tick
        slewtick=$[$tick * 91 / 100]
        duration=10.1
        echo correcting clock for inserted leap second...
        tickadj $slewtick
        sleep $duration
        tickadj $tick
        
  • Workaround 2:

    • stop ntpd
    • run ntptime -s 0 -f 0 to reset the kernel status and frequency
    • configure ntp with the -x option (ntp slew mode)
    • start ntpd (now in slew mode), have it running during the leap second.

In earlier versions of RHEL 5, RHEL 6, and RHEL 7 there were known issues with slew mode. See the following solutions documenting these issues:

  1. RHEL 5 - Time is not adjusted by a leap second when ntpd runs with -x option.
  2. RHEL 6 - Does Red Hat plan to release xleap.patch with ntp?
  3. RHEL 7 - Does Red Hat plan to release xleap.patch with ntp?

Note: RHEL 5 & RHEL6 (RHEL6 running kernel version lower than kernel-2.6.32-328.el6, as per Internal bugzilla ) systems with ntpd running without the -x flag will need to have ntpd stopped before June 30 0:00 UTC due to an issue clearing the kernel's leap status once it is set. RHEL 4 has ntp bug, even if we have enabled slew mode, it does not prevent ntpd from setting the kernel flag. More information can be find on article

In the first workaround the insertion of the leap second on the client will be ignored, and ntp can be safely restarted in regular mode after the leap second is inserted.

With the second workaround (slew mode '-x' ) the leap second will be ignored by kernel (NTPd will slowly adjust the time instead); however, any system running ntp-4.2.6p5-1.el6, ntp-4.2.6p5-2.el6_6 and ntp-4.2.6p5-19.el7_0 or below will have the leap second inserted (with stepping even if it is running in slew mode), but by ntpd instead of the kernel. It because of RHEL 6 bug 1199978 and RHEL 7 bug 1191122. On these versions ntpd will call either settimeofday() or clock_settime() at midnight to step the clock back automatically.

For those customers requiring accurate timestamp information, Red Hat has made available patched z-stream kernels for Red Hat Enterprise Linux 6.1, 6.2, and 6.3 that resolve any problem with leap seconds. These kernels must be installed before the next leap second event to address any known issues.

Red Hat Labs Testing

A lab has been made available to test for leap second vulnerabilities, and is available at Leap Second Issue Detector.

Testing the Leap Second Insertion

This program will insert an actual leap second and may cause issues on unpatched systems. It is strongly recommended to use this in a test environment.

A leap second stress test is attached to this article as leap-a-day.c, and may be downloaded and compiled on test systems. Instructions are below:

  1. Ensure that the kernel-headers package is installed:

    yum install kernel-headers
    
  2. Download the leap-a-day.c program attached to this article.

  3. Build the application locally:

    gcc leap-a-day.c -o leap-a-day -lrt
    
  4. Run the application:

    *  Usage: leap-a-day [-s] [-i <num>]
    *
    *  Options:
    *      -s:     Each iteration, set the date to 10 seconds before midnight GMT.
    *              This speeds up the number of leapsecond transitions tested,
    *              but because it calls settimeofday frequently, advancing the
    *              time by 24 hours every ~16 seconds, it may cause application
    *              disruption.
    *
    *      -i:     Number of iterations to run (default: infinite)
    *
    *  Other notes: Disabling NTP prior to running this is advised, as the two
    *               may conflict in thier commands to the kernel.
    

The test will alternate between testing insertions and deletions and will report any errors detected. This test will adjust the time of the system, so plan accordingly. Both a successful and failure run of an insertion appears below:

Note: For RHEL 4 and RHEL 5 systems only the first interval is valid. Subsequent runs of this test will be necessary for testing additional leap second insertions.

  • Success:

    # ./leap-a-day -s
    Setting time to speed up testing
    This runs continuously. Press ctrl-c to stop
    
    Setting time to Tue Mar 10 19:59:50 2015
    Scheduling leap second for Tue Mar 10 20:00:00 2015
    Sun Mar  8 19:59:57 2015 +    190 us (0)        TIME_INS
    Sun Mar  8 19:59:57 2015 + 500395 us (0)        TIME_INS
    Sun Mar  8 19:59:58 2015 +    604 us (0)        TIME_INS
    Sun Mar  8 19:59:58 2015 + 500808 us (0)        TIME_INS
    Sun Mar  8 19:59:59 2015 +   1011 us (0)        TIME_INS
    Sun Mar  8 19:59:59 2015 + 501219 us (0)        TIME_INS
    Sun Mar  8 19:59:59 2015 +   5731 us (1)        TIME_OOP
    Sun Mar  8 19:59:59 2015 + 505935 us (1)        TIME_OOP
    Sun Mar  8 20:00:00 2015 +   6136 us (1)        TIME_WAIT
    Sun Mar  8 20:00:00 2015 + 506345 us (1)        TIME_WAIT
    Sun Mar  8 20:00:01 2015 +   6551 us (1)        TIME_WAIT
    Sun Mar  8 20:00:01 2015 + 506678 us (1)        TIME_WAIT
    Sun Mar  8 20:00:02 2015 +   6881 us (1)        TIME_WAIT
    Leap complete    
    
  • Failure:

    # ./leap-a-day -s
    Setting time to speed up testing
    This runs continuously. Press ctrl-c to stop
    
    Setting time to Sun Mar  8 19:59:50 2015
    Scheduling leap second for Sun Mar  8 20:00:00 2015
    Sun Mar  8 19:59:57 2015 +    190 us (0)        TIME_INS
    Sun Mar  8 19:59:57 2015 + 500395 us (0)        TIME_INS
    Sun Mar  8 19:59:58 2015 +    604 us (0)        TIME_INS
    Sun Mar  8 19:59:58 2015 + 500808 us (0)        TIME_INS
    Sun Mar  8 19:59:59 2015 +   1011 us (0)        TIME_INS
    Sun Mar  8 19:59:59 2015 + 501219 us (0)        TIME_INS
    Sun Mar  8 19:59:59 2015 +   5731 us (1)        TIME_OOP
    Sun Mar  8 19:59:59 2015 + 505935 us (1)        TIME_OOP
    Sun Mar  8 20:00:00 2015 +   6136 us (1)        TIME_WAIT
    Sun Mar  8 20:00:00 2015 + 506345 us (1)        TIME_WAIT
    Sun Mar  8 20:00:01 2015 +   6551 us (1)        TIME_WAIT
    Sun Mar  8 20:00:01 2015 + 506678 us (1)        TIME_WAIT
    Sun Mar  8 20:00:02 2015 +   6881 us (1)        TIME_WAIT
    ERROR: hrtimer early expiration failure observed.
    Leap complete 
    

Attachments

  • Component
  • ntp

47 Comments

We run ntpd with the `-x` option, and were doing so over the June 2012 leap second, but we still saw issues caused by that leap second being applied.

It appears that ntpd notifies the kernel of the leap second regardless of the `-x` argument, and so [all the bugs](https://access.redhat.com/knowledge/solutions/173693) exposed by the June 2012 leap second will still be hit regardless of whether ntpd has the `-x` argument.

(This makes sense from a purist point of view: during a leap second, ntpd doesn't step the time, it simply informs the kernel of the leap second and lets the kernel deal with it. That the kernel deals with it by triggering a time jump isn't ntpd's business.)

Thus the second workaround proposed in this article, sadly, won't work. It'd have saved us a fair bit of pain if it had.

[much documentation here has been rewritten in the meantime, and much code has also changed]
When ntpd runs in slew mode, the kernel discipline is disabled and ntpd is doing the handling, so sleweing instead of stepping the time. If you see different behaviour, we are very interested in the details in a customer center case.
Details for reproducers to have systems undergo the leapsecond situation are here in kbase.

Why do we check for INS or DEL? What do we do if we see INS or DEL?

INS - corresponds to the leap second insertion
DEL - corresponds to the leap second deletion

While both are checked through the course of this test only leap second insertions have occurred so far (even though both are possible). You should see both over the course of this test, but the primary thing to check for is the following message:

ERROR: hrtimer early expiration failure observed.

If this message appears then an update is required.

I do not know how to run

[root@smartguy ~]$ ntpq -c "lassoc" -c "mrv &1 &999 leap,srcadr,stratum" name1.test.com | grep -E "leap=([^0].|.[^0])"
srcadr=test.cites.example.edu, leap=01, stratum=1
srcadr=time-b.nist.gov, leap=11, stratum=1

So if I understand correctly. "leap-a-day" is actually testing if a system is vulnerable to the panic that would occur during the real leap second insertion with ntpd running. It is not for testing one of the workarounds. Correct? The fact that ntpd needs to be off is just due to conflicts in the kernel between ntpd and "leap-a-day".

How is this test which checks to see if hrtimer has a particular failure, related to the original bug where the system hangs while printing the leap second insertion message?

In my environment, i switch ntp clients to slew mode and restart client. Do i still need to run un ntptime -s 0 command ?

assID=53081 status=9014 reach, conf, 1 event, event_reach,
srcadr=LOCAL(0), srcport=123, dstadr=127.0.0.1, dstport=123, leap=00

It is not required if you are putting the ntp in slew mode atleast 24 hours prior. If not, then use ntptime and then set it to slew mode

Ranjith
I stopped my ntp on my clients that I can not update my kernel's on, on the evening of 27 June CDT. with i run ntpttime I see the INS.

does this mean I have an INS leap second ecord coming in.
should i run ntptime -s 0 to clear it.

Larry,

If you notice INS in the output(pasting an example output)then you should reset the flag using ntptime -s 0 - f 0

“status 0x2011 (PLL,INS,NANO))”.

Thanks Ranjith
i have 4.7 i have after running ntptime -s 0 -f 0
status 0x0 (),
then run ntptime again to double check
status 0x40 (UNSYNC),
So i should be good.

There was an INS in the status line last night when I submitted the comment. and I had seen another article from redhat that had the two work around's Just wanted to double check with the expert on this.

Thanks again.

Hi Ranjith,
Even after running "ntptime -s 0 - f 0" we are seeing the INS flag. would appreciate if any advises.
Server is RHEL 4.5 , with latest ntp package recommended by redhat. Kernel version less than recommended.
Linux serverxxxx 2.6.9-55.0.2.ELsmp #1 SMP Tue Jun 12 17:59:08 EDT 2007 i686 i686 i386 GNU/Linux

Before

[root@serverxxxx ]# ntptime
ntp_gettime() returns code 1 (INS)
time d93ccad9.6723a000 Tue, Jun 30 2015 16:59:21.402, (.402887),
maximum error 478592 us, estimated error 135 us
ntp_adjtime() returns code 1 (INS)
modes 0x0 (),
offset 220.000 us, frequency 156.567 ppm, interval 4 s,
maximum error 478592 us, estimated error 135 us,
status 0x11 (PLL,INS),
time constant 6, precision 1.000 us, tolerance 512 ppm,
pps frequency 0.000 ppm, stability 512.000 ppm, jitter 200.000 us,
intervals 0, jitter exceeded 0, stability exceeded 0, errors 0.

[root@serverxxxx ]# /sbin/service ntpd stop
ntpd を停止中: [ OK ]
[root@serverxxxx ]#
[root@serverxxxx ]# ntptime -s 0 -f 0
ntp_gettime() returns code 1 (INS)
time d93ccb17.f894b000 Tue, Jun 30 2015 17:00:23.971, (.971019),
maximum error 510336 us, estimated error 135 us
ntp_adjtime() returns code 1 (INS)
modes 0x12 (FREQUENCY,STATUS),
offset 217.000 us, frequency 0.000 ppm, interval 4 s,
maximum error 510336 us, estimated error 135 us,
status 0x0 (),
time constant 6, precision 1.000 us, tolerance 512 ppm,
pps frequency 0.000 ppm, stability 512.000 ppm, jitter 200.000 us,
intervals 0, jitter exceeded 0, stability exceeded 0, errors 0.

[root@serverxxxx ]# /sbin/service ntpd start
ntpd: 時間サーバと同期中: [ OK ]
ntpd を起動中: [ OK ]

[root@serverxxxx ]# ntptime
ntp_gettime() returns code 1 (INS)
time d93ccb43.a8771000 Tue, Jun 30 2015 17:01:07.658, (.658067),
maximum error 12816 us, estimated error 16 us
ntp_adjtime() returns code 1 (INS)
modes 0x0 (),
offset 0.000 us, frequency 156.565 ppm, interval 4 s,
maximum error 12816 us, estimated error 16 us,
status 0x1 (PLL),
time constant 0, precision 1.000 us, tolerance 512 ppm,
pps frequency 0.000 ppm, stability 512.000 ppm, jitter 200.000 us,
intervals 0, jitter exceeded 0, stability exceeded 0, errors 0.

[root@serverxxxx ]# ntptime
ntp_gettime() returns code 5 (ERROR)
time d93ccc06.68c2b000 Tue, Jun 30 2015 17:04:22.409, (.409221),
maximum error 17712 us, estimated error 523 us
ntp_adjtime() returns code 5 (ERROR)
modes 0x0 (),
offset 934.000 us, frequency 156.756 ppm, interval 4 s,
maximum error 17712 us, estimated error 523 us,
status 0x41 (PLL,UNSYNC),
time constant 2, precision 1.000 us, tolerance 512 ppm,
pps frequency 0.000 ppm, stability 512.000 ppm, jitter 200.000 us,
intervals 0, jitter exceeded 0, stability exceeded 0, errors 0.

After approx 20 mins

[root@serverxxxx ]# ntptime
ntp_gettime() returns code 1 (INS)
time d93ccd4f.5ec59000 Tue, Jun 30 2015 17:09:51.370, (.370202),
maximum error 493339 us, estimated error 521 us
ntp_adjtime() returns code 1 (INS)
modes 0x0 (),
offset 133.000 us, frequency 156.843 ppm, interval 4 s,
maximum error 493339 us, estimated error 521 us,
status 0x11 (PLL,INS),
time constant 2, precision 1.000 us, tolerance 512 ppm,
pps frequency 0.000 ppm, stability 512.000 ppm, jitter 200.000 us,
intervals 0, jitter exceeded 0, stability exceeded 0, errors 0.
[root@serverxxxx ]#
[root@serverxxxx ]# ps -ef | grep ntp
ntp 11445 1 0 17:00 ? 00:00:00 ntpd -x -u ntp:ntp -p /var/run/ntpd.pid
root 12298 11204 0 17:10 pts/0 00:00:00 grep ntp
[root@serverxxxx ]#

Hi,

For RHEL 4 and RHEL5 servers, ntptime -s0 -f0 does not reset the kernel flag. You will have to reboot the server if the kernel flag is already set. Once the flag is set, then there is no point in configuring ntp in slew mode. If you have any further questions, please contact Red Hat Technical Support

The above note is already documented in this article.

In that case can we simply stop NTP and start it post leap second event? Will stopping NTP clear the INS flag in "ntptime"?

No, ntptime is used to clear the kernel flag. stopping ntp will not clear the once set kernel ntp leap flag

So the bottomline is for RHEL 4/5 servers with lower kernel version and ntptime returing INS flag, none of the workaround (stopping ntpd and slew mode) is effective. Reboot is the only option to clear teh flag and then stop ntpd? or Server will be vulnerable to the aftermath of leap second insertion?

If the flag is set, reboot is the only option to clear it. None of the workarounds will help. Before reboot, do a chkonfig ntpd off. After reboot, you can wait for the leap second to pass by or start ntpd in slew mode. For RHEL4, slew mode configuration seems not working as shown in the above comments. Need to verify. Assuming that you are starting ntp after leap second has passed by, the system will consider a one second offset with the ntp server and then steps up the time. it will not consider this as leap second event but a normal step up. System should not be vulnerable and should be working normally until next leap second event( if the packages are not patched, you might notice the same issue again. Next leap second event has not been announced)

Hi Ranjith,

NTP package was upgraded and slew mode was set couple of weeks before.
we did the steps #ntptime -s0 -f0 because
1. To avoid OS crash due to https://access.redhat.com/solutions/1325313
2. Since kernel version was less than recommended and we went for work around with ntptime command.
3. It was consulted with Redhat with a case , no reboot info was mentioned.

ntptime seems to have worked for me on redhat 4.7

cat /etc/redhat-release

Red Hat Enterprise Linux AS release 4 (Nahant Update 7)

./leap_vulnerability.sh

Installed kernel version: 2.6.9-78.0.8.ELsmp
Installed tzdata version: tzdata-2015d-1.el4
Installed ntp version: ntp-4.2.0.a.20040617-8.el4

[SUGGESTIONS]
The running kernel is vulnerable to a performance degradation after the Leap Second Insertion of June 30, 2015.
Please refer to https://access.redhat.com/articles/15145 for remediation steps.

ntpq -c "lassoc" -c "mrv &1 &999 leap,srcadr,stratum" |grep leap

dstport=123, leap=01, stratum=2, precision=-19, rootdelay=27.100,
dstadr=10.129.188.15, dstport=123, leap=01, stratum=2, precision=-19,

root@coorlpdb02 [/root]

ntptime |grep status

status 0x11 (PLL,INS),

root@coorlpdb02 [/root]

service ntpd stop

Shutting down ntpd: [ OK ]

root@coorlpdb02 [/root]

ntptime |grep status

status 0x11 (PLL,INS),

ntptime -s0 -f0

ntp_gettime() returns code 1 (INS)
time d93d1ed7.505be000 Tue, Jun 30 2015 8:57:43.313, (.313902),
maximum error 312835 us, estimated error 150450 us
ntp_adjtime() returns code 1 (INS)
modes 0x12 (FREQUENCY,STATUS),
offset 26.000 us, frequency 0.000 ppm, interval 4 s,
maximum error 312835 us, estimated error 150450 us,
status 0x0 (),
time constant 2, precision 1.000 us, tolerance 512 ppm,
pps frequency 0.000 ppm, stability 512.000 ppm, jitter 200.000 us,
intervals 0, jitter exceeded 0, stability exceeded 0, errors 0.

root@coorlpdb02 [/root]

ntptime |grep status

status 0x0 (),

Larry,

Even though the ntptime does not list INS, the kernel will step the Clock in RHEL 4.

Edit:
Just to add more,
Once the INS flag is set, kernel will step the clock by 1 second. ntptime -s0 -f0 will show as if it cleared the flag but actually it does not.

Ranjith
Is there another way to check for the INS/DEL flag without running ntpq which requires ntpd to be running?

larry,

[root@localhost ~]# ntptime
ntp_gettime() returns code 1 (INS)
time d93da7a1.ce77e000 Tue, Jun 30 2015 23:41:21.806, (.806517),
maximum error 910267 us, estimated error 217725 us
ntp_adjtime() returns code 1 (INS)
modes 0x0 (),
offset -2914.000 us, frequency -336.501 ppm, interval 4 s,
maximum error 910267 us, estimated error 217725 us,
status 0x11 (PLL,INS), <------------------------------------------------- INS mean insert leap second.
time constant 0, precision 1.000 us, tolerance 512 ppm,
pps frequency 0.000 ppm, stability 512.000 ppm, jitter 200.000 us,
intervals 0, jitter exceeded 0, stability exceeded 0, errors 0.

Thanks Rupesh
On RHEL 4 I had one server that i had not stopped ntpd over the weekend.
I ran ntpq -c ..... saw the leap=01
stopped ntpd
did the ntptime -s0 -f0
No longer see the INS when running ntptime
but according to Ranjith
Even though the ntptime does not list INS, the kernel will step the Clock in RHEL 4
with ntpd down i wanted to see if there was another way to see the leap=01 kernel flag

Hello Larry,

We had test it internally in our test systems and observed the same. We cleared information through ntptime command and we did not see any more flags but kernel still insert leap second.

Hi, I have few RedHat 4 hosts..
cat /etc/redhat-release
Red Hat Enterprise Linux AS release 4 (Nahant Update 6)

we dont have the redhat ELS subscriptionand no way to update the packages for tzdat or ntp... any suggestions on how to oversome tonights leapsecond issue onthese redhat 4 servers..
2.6.9-67.0.20.EL
ntp-4.2.0.a.20040617-6.el4
tzdata-2008b-3.el4

You can verify if ntp passed leap second information to the kernel or not. If it has been set, we do not have any option other then reboot. Turn off ntp using chkconfig, reboot the system and run it till leap second passed.

Command to verify leap second flag is #ntptime. Check status in command output. If kernel is aware of leap second it must show you 'INS' in status.

So the servers I shut ntp off on 27 and do not show INS in the status line those should be ok?

Thanks Rupesh for the info..yeah i am seeing INS in the status of #ntpdate o/p..but do we have to perform the reboot prior to the leap? i.e. 8:00PM EST..

ntp_gettime() returns code 1 (INS)
time d93d4540.9af32000 Tue, Jun 30 2015 12:41:36.605, (.605272),
maximum error 245493 us, estimated error 989 us
ntp_adjtime() returns code 1 (INS)

and is it he same procedure have to follow for rhwl 5 and 6 ..

i have the below redhat servers with kernel mentioned versions available.
RHEL-5.3 --2.6.18-128.el5
RHEL-5.11 --2.6.18-404.el5
RHEL-6.4 --2.6.32-358.el6.x86_64
RHEL-6.5 --2.6.32-431.17.1.el6.x86_64
RHEL-6.6 --2.6.32-504.8.1.el6.x86_64
modes 0x0 (),
offset -5220.000 us, frequency 126.487 ppm, interval 4 s,
maximum error 245493 us, estimated error 989 us,
status 0x11 (PLL,INS),
time constant 6, precision 1.000 us, tolerance 512 ppm,
pps frequency 0.000 ppm, stability 512.000 ppm, jitter 200.000 us,
intervals 0, jitter exceeded 0, stability exceeded 0, errors 0.

Same procedure for RHEL 5. For RHEL 6 we have time and do not require reboot. Have you open case with us? we can send more details on it.

yeah here is the case # 01469195

Hi, I can see my colleague has updated the case. You have any query in that?

yeah like do we have to reboot the RHEL- 4 and 5 service prior to the leap?

If you see flags are already set in kernel, yeah you would need to reboot the node (turn off ntpd first and reboot and run ntp free system till leap second passed). In RHEL 4/5 there is no option to reset that information once it set.

Yes, you will have to reboot the RHEL4 and RHEL5 prior to leap. Please make sure to set chkconfig ntpd off before the reboot. RHEL 5 can be configured in slew mode after the reboot. For RHEL4, you have to keep the ntp service in stopped state until the leap event passes by or update the kernel to minimum kernel-2.6.9-89.EL so that you can avoid the known kernel hang issues when the leap second is inserted

The Leap Second Issue Detector Red Hat Access Labs app has been updated to detect the upcoming leap second event that will occur on 2016 December 31, 23h 59m 60s UTC. https://access.redhat.com/labs/leapsecond/

this script only handles rhel 6 & 7 not 4 & 5

Have you tried the script on RHEL 5 ?. I have reported an error while running the script on RHEL 4

Are you facing any error while running the script on RHEL 5 ?

From my test RHEL 5 system

[INFORMATION] - Installed kernel version: 2.6.18-371.el5 - The system is running NTP: ntp-4.2.2p1-18.el5_11.x86_64 When the leap second occurs, this systems time will be stepped by the kernel. Thus, it is configured to stay in sync with the true/official time.

If you would like to learn more on how to resolve Leap Second Issues in Red Hat Enterprise Linux, refer to https://access.redhat.com/articles/15145.

I have tried it on rhel4 and yes there is an error. on rhel 5 it complains about the kernel which i expected because the server i am checking, the oracle dba's do not want the kernel updated because it affects their asm disk cluster. but for tzdata and ntp it does not complain even though the tzdata and ntp packages are old. coefsdb01 ~]$ rpm -q tzdata tzdata-2010e-1.el5 tzdata-2015d-1.el5 coefsdb01 ~]$ rpm -q ntp ntp-4.2.2p1-9.el5_4.1 I also opened a case with redhat - to help determine the relative vs absolute timer applications

Dear team, Can you please help me to understand whether i need to take any action in case i am getting below output -

For OS: RHEL 7 7.2 (Maipo) below is result ]# ntpq -c "lassoc" -c "mrv &1 &999 leap,srcadr,stratum"

ind assid status conf reach auth condition last_event cnt

1 36554 941a yes yes none candidate sys_peer 1 2 36555 961a yes yes none sys.peer sys_peer 1 srcadr=ntp1.server.com, leap=00, stratum=4 srcadr=ntp2.server.com, leap=00, stratum=4

#

For OS: RHEL6 below is results. ]# sh leap_second_issue_detector.sh [INFORMATION] - Installed kernel version: 2.6.32-642.6.1.el6.x86_64 - The system is running NTP: ntp-4.2.6p5-10.el6.1.x86_64 When the leap second occurs, this systems time will be stepped by the kernel. Thus, it is configured to stay in sync with the true/official time. If you would like to learn more on how to resolve Leap Second Issues in Red Hat Enterprise Linux, refer to https://access.redhat.com/articles/15145.

#

For OS RHEL5 Below is the result

sh leap_second_issue_detector.sh

[INFORMATION] - Installed kernel version: 2.6.18-416.el5 - The system is running NTP: ntp-4.2.2p1-18.el5_11.x86_64 When the leap second occurs, this systems time will be stepped by the kernel. Thus, it is configured to stay in sync with the true/official time. If you would like to learn more on how to resolve Leap Second Issues in Red Hat Enterprise Linux, refer to https://access.redhat.com/articles/15145. #

#

For OS: RHEL4 below is the results.

sh leap_second_issue_detector.sh

[INFORMATION] - Installed kernel version: 2.6.9-89.ELsmp After the leap second, this system will have a time which is different from the true/official time. If that is not desired, consult https://access.redhat.com/articles/15145 for details on configuring the system.

NOTE: I want to know if above mentioned is the situation in my environment do i need to take any action to avoid disruptions to the application/Database/servers due to leap second insertions ?

It would be advisable to open a support request for these sort of detailed questions 1. For RHEL7: leap flag is 00. Leap flag is no yet set 2. For RHEL6: NTP/Kernel are the latest version and covers all the known issues. Leapsecond will be inserted or kernel will step back the clock on Dec 31 midnight. No action required 3. For RHEL5: Not action required. 4. For RHEL4: no ntp detected. There will be no leap second inserted on the system

Hi!

When will the ntp server flag the leap second so that the kernel bit will be set?

[root@ ~]# ./leap_second_issue_detector.sh

[INFORMATION]

  • Installed kernel version: 2.6.32-220.4.2.el6.i686

  • The system is running NTP: ntp-4.2.4p8-2.el6.i686

When the leap second occurs, this systems time will be stepped by the kernel. Thus, it is configured to stay in sync with the true/official time.

[SUGGESTIONS ON KERNEL] Known issues of kernel are detected and listed below. Refer to the link attached for the remediation steps.

[root@ ~]# ntpq -c "lassoc" -c "mrv &1 &999 leap,srcadr,stratum" | grep -i 'leap'

dstport=123, leap=00, stratum=3, precision=-24, rootdelay=1.923,

dstport=123, leap=00, stratum=3, precision=-24, rootdelay=1.984,

[root@ ~]# ntptime |egrep 'INS|DEL'

[root@ ~]#

Rgds Christian

->>When will the ntp server flag the leap second so that the kernel bit will be set?

As per the leapsecond issue detector script output, NTP version is 4.2.4:. For 4.2.4 If one of the NTP server announces leap second, NTP client will accept it and set the kernel flag on the last day of June/Dec

When can you expect the ntp server to announce the leap second?

Will it not set the kernel bit/flag immediately once the leap second is announced?

I think I saw the leap second being announced on one of our systems the other day (leap=01) but now it's gone again (leap=00). Can that be possible? The kernel bit/flag is not set on that system as per the test "ntptime |egrep 'INS|DEL'".

Rgds Christian

->>When can you expect the ntp server to announce the leap second?

NTP server is expected to announce the leapsecond any time. NTP server in your organization should get the announcement from its upstream. There is no fixed time.. some of them have already announced(as far as I know)

->>Will it not set the kernel bit/flag immediately once the leap second is announced?

It does not. ntp client based on 4.2.4 will only set the kernel flag on the last day of June/Dec

->>I think I saw the leap second being announced on one of our systems the other day (leap=01) but now it's gone again (leap=00). Can that be possible?

It would be good if you can you open a support request. Is the ntp client based on 4.2.6 where you noticed it ?. Leapsecond notification can be disarmed if the majority of upstream server does not agree on the leapsecond

But shouldn't this test "ntptime |egrep 'INS|DEL'" show immediately that the leap second flag/bit has been set, after it has been announced, even if it does not trigger until last day of Dec?

Rgds Christian

Announcement is received by the NTP client on any day(leap=01)

NTP(4.2.4) client only sets the kernel flag "ntptime |egrep 'INS|DEL'" on the last day of June or Dec. NTP (42.6) sets the kernel flag at the end of any month.