How to clear the Leap Second Insertion flag after it has been received?

Updated -

Introduction

For the primary documentation on leap seconds, including a list of known issues, see Resolve Leap Second Issues in Red Hat Enterprise Linux.

While the RHEL kernel accepts a leapsecond for any month, ntpd in RHEL does not hand over the leapsecond in all cases. The ntp version ntp-4.2.4p8 sets the kernel leap bit only on end of June or December. The leap information will be forwarded to the clients on any day though. ntp Versions 4.2.6 and later set the kernel leap bit on end of any month, but require that a majority of its servers agree on the leap instead of only one as in 4.2.4 and before.

NTP servers can issue a leap second at the end of any month. You should check your ntp servers to see if they are aligned to issue a leap second:

[root@host ~]# ntpq -c "lassoc" -c "mrv &1 &999 leap,srcadr,stratum"

When you see leap=01, you have a leap second coming your way. Probably good idea to start checking at the end of every month and currently imperative to check at the end of the months of June and December.

To verify if the local system has currently leapsecond bits set:

[root@host ~]# ntptime |egrep 'INS|DEL'
ntp_gettime() returns code 1 (INS)
ntp_adjtime() returns code 1 (INS)
  status 0x2011 (PLL,INS,NANO),

Preventing Issues and Workarounds

First, see if you need to update Leap Second Issue Detector Red Hat Labs Tool

Red Hat strongly recommends that customers update their systems to the most recent available Red Hat Enterprise kernels. That said, however, the following two NTP client workarounds may be used with any systems that require precise, but not accurate, timestamp information. (In this context, "precise" means "a measure of the detail in which a quantity is expressed." "Accurate" means the "measurement deviation from true value.")

  • Workaround 1:

    • stop ntpd
    • run ntptime -s 0 to reset the kernel status
    • run ntptime and verify it doesn't print INS or DEL on the status line
    • run the system now without ntp active during the leap second

    • at some point in the future you might want to sync the systems idea of time again with the rest of the world. These methods could be used:

      • stepping the clock: when running ntpd again, the clock will be stepped
      • correct the clock slowly: ntpdate -B <some-ntp-server> could be used, the correction takes about 2000 seconds. After that, ntpd can be started again and the error will be already corrected, so ntpd won't need to perform any steps.

      • correct the clock faster: This uses the adjtimex tick value. The following script slows down the clock by 9% for 10.1 seconds. It can be used instead of ntpdate in the workaround I described earlier:

        #!/bin/bash
tick=$(tickadj | awk '{print $3}')
echo initial tick $tick
slewtick=$[$tick * 91 / 100]
duration=10.1
echo correcting clock for inserted leap second...
tickadj $slewtick
sleep $duration
tickadj $tick

  • Workaround 2:

    • stop ntpd
    • run ntptime -s 0 -f 0 to reset the kernel status and frequency
    • configure ntp with the -x option (ntp slew mode)
    • start ntpd (now in slew mode), have it running during the leap second.

In earlier versions of RHEL 5, RHEL 6, and RHEL 7 there were known issues with slew mode. See the following solutions documenting these issues:

  1. RHEL 5 - Time is not adjusted by a leap second when ntpd runs with -x option.
  2. RHEL 6 - Does Red Hat plan to release xleap.patch with ntp?
  3. RHEL 7 - Does Red Hat plan to release xleap.patch with ntp?

Note: RHEL 5 & RHEL6 (RHEL6 running kernel version lower than kernel-2.6.32-328.el6, as per Internal bugzilla ) systems with ntpd running without the -x flag will need to have ntpd stopped before June 30 0:00 UTC due to an issue clearing the kernel's leap status once it is set. RHEL 4 has ntp bug, even if we have enabled slew mode, it does not prevent ntpd from setting the kernel flag. More information can be find on article

In the first workaround the insertion of the leap second on the client will be ignored, and ntp can be safely restarted in regular mode after the leap second is inserted.

With the second workaround (slew mode '-x' ) the leap second will be ignored by kernel (NTPd will slowly adjust the time instead); however, any system running ntp-4.2.6p5-1.el6, ntp-4.2.6p5-2.el6_6 and ntp-4.2.6p5-19.el7_0 or below will have the leap second inserted (with stepping even if it is running in slew mode), but by ntpd instead of the kernel. It because of RHEL 6 bug 1199978 and RHEL 7 bug 1191122. On these versions ntpd will call either settimeofday() or clock_settime() at midnight to step the clock back automatically.

For those customers requiring accurate timestamp information, Red Hat has made available patched z-stream kernels for Red Hat Enterprise Linux 6.1, 6.2, and 6.3 that resolve any problem with leap seconds. These kernels must be installed before the next leap second event to address any known issues.

Red Hat Labs Testing

A lab has been made available to test for leap second vulnerabilities, and is available at Leap Second Issue Detector.

Testing the Leap Second Insertion

This program will insert an actual leap second and may cause issues on unpatched systems. It is strongly recommended to use this in a test environment.

A leap second stress test is attached to this article as leap-a-day.c, and may be downloaded and compiled on test systems. Instructions are below:

  1. Ensure that the kernel-headers package is installed:

    yum install kernel-headers

  2. Download the leap-a-day.c program attached to this article.

  3. Build the application locally:

    gcc leap-a-day.c -o leap-a-day -lrt

  4. Run the application:

    *  Usage: leap-a-day [-s] [-i <num>]
*
*  Options:
*      -s:     Each iteration, set the date to 10 seconds before midnight GMT.
*              This speeds up the number of leapsecond transitions tested,
*              but because it calls settimeofday frequently, advancing the
*              time by 24 hours every ~16 seconds, it may cause application
*              disruption.
*
*      -i:     Number of iterations to run (default: infinite)
*
*  Other notes: Disabling NTP prior to running this is advised, as the two
*               may conflict in thier commands to the kernel.

The test will alternate between testing insertions and deletions and will report any errors detected. This test will adjust the time of the system, so plan accordingly. Both a successful and failure run of an insertion appears below:

Note: For RHEL 4 and RHEL 5 systems only the first interval is valid. Subsequent runs of this test will be necessary for testing additional leap second insertions.

  • Success:

    # ./leap-a-day -s
Setting time to speed up testing
This runs continuously. Press ctrl-c to stop

Setting time to Tue Mar 10 19:59:50 2015
Scheduling leap second for Tue Mar 10 20:00:00 2015
Sun Mar  8 19:59:57 2015 +    190 us (0)        TIME_INS
Sun Mar  8 19:59:57 2015 + 500395 us (0)        TIME_INS
Sun Mar  8 19:59:58 2015 +    604 us (0)        TIME_INS
Sun Mar  8 19:59:58 2015 + 500808 us (0)        TIME_INS
Sun Mar  8 19:59:59 2015 +   1011 us (0)        TIME_INS
Sun Mar  8 19:59:59 2015 + 501219 us (0)        TIME_INS
Sun Mar  8 19:59:59 2015 +   5731 us (1)        TIME_OOP
Sun Mar  8 19:59:59 2015 + 505935 us (1)        TIME_OOP
Sun Mar  8 20:00:00 2015 +   6136 us (1)        TIME_WAIT
Sun Mar  8 20:00:00 2015 + 506345 us (1)        TIME_WAIT
Sun Mar  8 20:00:01 2015 +   6551 us (1)        TIME_WAIT
Sun Mar  8 20:00:01 2015 + 506678 us (1)        TIME_WAIT
Sun Mar  8 20:00:02 2015 +   6881 us (1)        TIME_WAIT
Leap complete

  • Failure:

    # ./leap-a-day -s
Setting time to speed up testing
This runs continuously. Press ctrl-c to stop

Setting time to Sun Mar  8 19:59:50 2015
Scheduling leap second for Sun Mar  8 20:00:00 2015
Sun Mar  8 19:59:57 2015 +    190 us (0)        TIME_INS
Sun Mar  8 19:59:57 2015 + 500395 us (0)        TIME_INS
Sun Mar  8 19:59:58 2015 +    604 us (0)        TIME_INS
Sun Mar  8 19:59:58 2015 + 500808 us (0)        TIME_INS
Sun Mar  8 19:59:59 2015 +   1011 us (0)        TIME_INS
Sun Mar  8 19:59:59 2015 + 501219 us (0)        TIME_INS
Sun Mar  8 19:59:59 2015 +   5731 us (1)        TIME_OOP
Sun Mar  8 19:59:59 2015 + 505935 us (1)        TIME_OOP
Sun Mar  8 20:00:00 2015 +   6136 us (1)        TIME_WAIT
Sun Mar  8 20:00:00 2015 + 506345 us (1)        TIME_WAIT
Sun Mar  8 20:00:01 2015 +   6551 us (1)        TIME_WAIT
Sun Mar  8 20:00:01 2015 + 506678 us (1)        TIME_WAIT
Sun Mar  8 20:00:02 2015 +   6881 us (1)        TIME_WAIT
ERROR: hrtimer early expiration failure observed.
Leap complete

Attachments

  • Component
  • ntp

Comments