Warning message

Log in to add comments or rate this document.

Resolve Leap Second Issues in Red Hat Enterprise Linux

Updated 2015-04-22T12:40:54+00:00

Contents

  1. Introduction
  2. Upcoming Leap Second Events
  3. Environment
  4. Understanding the Leap Second
  5. Handling of the Leap Second
  6. Known Issues
  7. Testing
  8. Additional Resources

Introduction

Leap seconds are a periodic one-second adjustment of Coordinated Universal Time(UTC) in order to keep a system's time of day close to the mean solar time. However, the Earth's rotation speed varies in response to climatic and geological events, and due to this, UTC leap seconds are irregularly spaced and unpredictable. This article seeks to provide information regarding the leap seconds themselves, along with addressing how Red Hat Enterprise Linux handles the insertion of one.

Upcoming Leap Second Events

The next leap second insertion is scheduled for June 30th, 2015 at 23:59:60 UTC.

Environment

  • Red Hat Enterprise Linux versions 4
  • Red Hat Enterprise Linux versions 5
  • Red Hat Enterprise Linux versions 6
  • Red Hat Enterprise Linux versions 7

Understanding the Leap Second

The basic time for mostly all of the world's local time zones is called Coordinated Universal Time, UTC, which is derived from a bunch of atomic clocks which are distributed in several countries all over the world. The rotation of the earth is not very constant and varies a bit over time, while decreasing the mean rotation speed slowly. This is the reason why so called leap seconds are inserted into the UTC time scale, they adjust process of the UTC time to the real earth rotation.

Why this extra second? It exists because the rotation of the Earth on its axis, which determines the passing of days and nights, slows down over a long period, mainly as a consequence of Moon-Sun attraction effects. In addition, the Earth is affected by its internal (core, mantle) and external (atmosphere, oceans) constituents. Nowadays, though, time is measured largely by 250 atomic clocks belonging to several countries, which function by measuring the transition of energy levels in an atom. Together these clocks are used to calculate UTC, and as this time measuring mechanism is independent of the Earth periodically the two must be brought into alignment with a leap second. In addition, we have to consider that the length of the day is nowadays 2 ms longer than in the year 1820. Not surprisingly then, the Earth's rotation slowly gets out of synchronization with UTC.

The International Earth Rotation Service, IERS, is measuring the true earth rotation and determines when a leap second has to be inserted. Insertion of a leap second is always scheduled for the end of the last day of a month, preferably at the end of June or December, at UTC midnight. In the past all leap seconds had been inserted at either one of those times(*). Announcements whether a leap second is scheduled or not are published by the IERS in their Bulletin C. The current Bulletin C is published about half a year before the next possible date for a leap second.

The IERS Bulletin C #30 from July 2005 announced a leap second to be inserted at the end of December 31, 2005, at UTC midnight. This was the first leap second that had been inserted since the end of 1998. This is why many applications which had been developed during the previous 7 years could not handle the leap second correctly.

Since, the leap second was inserted at the same moment all over the world, the local (civil) time of the insertion depends on the local time offset from UTC, e.g. if the time zone is UTC +3h then the leap second will be inserted when the wall clocks show 3 hours after midnight.

The standard way to count UTC time across a leap second is:

  2011-12-31 23.59.57
  2011-12-31 23.59.58
  2011-12-31 23.59.59
  2011-12-31 23.59.60 <-- leap second
  2012-01-01 00.00.00
  2012-01-01 00.00.01
  2012-01-01 00.00.02

If the system clock is kept in TAI and a right/* timezone is used then 23:59:60 may be listed; however, as 23:59:60 does not exist in Unix's implementation of UTC then the linux kernel inserts the leap second by stepping the system clock back by one second on the first clock update after 0:00 UTC. In local time counting depends on the time zone offset, e.g. for UTC+3h you would observe the following:

2012-01-01  02:58:00.000
2012-01-01  02:58:00.500
2012-01-01  02:59:00.000
2012-01-01  02:59:00.500
2012-01-01  03:00:00.0xx <-- leap second inserted
2012-01-01  02:59:00.0xx
2012-01-01  02:59:00.500
2012-01-01  03:00:00.000
2012-01-01  03:00:00.500

Leap seconds are a discontinuity of civil time. The time does not continue to increase monotonically but it is stepped by one second. Let's have a look at the time stamps of a leap second which is inserted, and the second after that leap second:

  2011-12-31 23.59.60 <-- leap second
  2012-01-01 00.00.00

We can normalize the time and date of the leap second:
60 seconds are 1 minute, which lets the minutes increment from 59 to 60
60 minutes are 1 hour, which lets the hours increment from 23 to 24
24 hours are 1 day, which lets the date increment, and so on.
Finally we can say that both lines represent exactly the same time, or 2 consequent seconds have the same time stamp.

Several time dissemination services also propagate the announcement of a leap second after this has been determined by the IERS. This includes for example the German long wave transmitter DCF77 and the satellite based navigation system GPS, so receivers which decode the signals from those systems can also decode the leap second announcement. Applications which read the time from those receivers can also determine the leap second announcement if this information is included in the applied protocol, e.g the time string transmitted by the receiver.

Please note that time code receivers can only pass the announcement of a leap second to the application, and count the time correctly over that period. It's the task of the application and/or operating system to handle that leap second correctly.

(*): An additional leap second notification was erroneously issued by some NTP (Network Time Protocol) servers within the North America region for midnight UTC on August 31, 2012. Two stratum 1 servers -- truechimer.cites.illinois.edu and time-b.nist.gov -- were identified as erroneously issuing the leap second notification. Any higher stratum (stratum 2, 3, 4....) NTP server with either of these two servers in their hierarchy should be considered as having issued the leap second incorrectly. NTP clients using these servers will set the kernel's leap indicator flag, which cannot be cleared until the system is rebooted.

Handling of the Leap Second

The handling of this issue depends on whether or not the Red Hat Enterprise Linux system is running a NTP (Network Time Protocol) or PTP (Precision Time Protocol) daemon.

Systems running NTP

Systems running any version of Red Hat Enterprise Linux should automatically account for leap second corrections if they are using the NTP (Network Time Protocol) daemon to synchronize their local timekeeping with an NTP server. During the last day before a leap second correction, NTP servers should notify their clients that a leap second will occur, and at 23:59:59 UTC, the Linux kernel should add or remove an extra second by making the 60th second occur twice or removing it entirely. Thus, Red Hat Enterprise Linux systems running an NTP client during the last leap second correction should have counted time as follows:

2008-12-31 23:59:59:052549000 UTC    <-- 1st occurrence of the 60th second
2008-12-31 23:59:59:259988000 UTC
2008-12-31 23:59:59:465214000 UTC
2008-12-31 23:59:59:669629000 UTC
2008-12-31 23:59:59:873936000 UTC
2008-12-31 23:59:59:079184000 UTC    <-- 2nd occurrence of the 60th second
2008-12-31 23:59:59:284011000 UTC
2008-12-31 23:59:59:488648000 UTC
2008-12-31 23:59:59:692691000 UTC
2008-12-31 23:59:59:896577000 UTC
2009-01-01 00:00:00:052378000 UTC

When the leap second occurs, the kernel prints a message to the system log. 

Systems running PTP

The timestamps exchanged in PTP are typically in TAI (International Atomic Time) which does not contain leap seconds; however, ptp4l and phc2sys will set the kernel flag to insert a leap second as the system clock continues to run in UTC. The kernel will then insert the leap second as normal.

Systems not running NTP or PTP

Note: Red Hat does not recommend running without a source of time synchronization enabled as there is no guarantee that systems will remain in sync with each other. Time drift may occur on each system as determined by its clock source. The decision to run without NTP/PTP/Chrony will need to be made on a per-system or per-cluster basis with the understanding that time drift may occur.

By default, Linux systems not using NTP or PTP to synchronize their timekeeping will not correct for leap seconds, and the time reported by these systems will have a one-second difference relative to UTC after the leap second correction. You should reset the clock manually after leap seconds occur.

You can also configure these systems to report time corrected for leap seconds by updating the tzdata package to the latest version available, copying the appropriate file from the /usr/share/zoneinfo/right directory hierarchy to /etc/localtime, and resetting the clock to the correct local time.  The files in /usr/share/zoneinfo/right contain local time information corrected for all leap seconds that have occurred since the beginning of the Epoch on 1970-01-01 00:00:00 UTC.  The other time zone files in /usr/share/zoneinfo do not have leap second corrections added.&nbsp. After the 2008 leap second, there were 24 leap seconds added since the Epoch. Any application that expects the time to be in UTC will have issues if a right/* timezone is used.

As an example, if a system is in the America/Los_Angeles (US Pacific) time zone, you can reconfigure the system to report leap-second-corrected time by running the following and resetting the clock to Pacific Time:

cp /usr/share/zoneinfo/right/America/Los_Angeles /etc/localtime

After /etc/localtime has been changed glibc will reload this file automatically; in addition, note that adjusting timezones does not change the system time, only the conversion from the system time to local time that happens in applications via glibc functions, such as localtime(), ctime(), etc. A system restart is not required after this update; however, if any applications are caching results from these functions then these applications may need to be restarted after tzdata has been updated.

Known Issues

Errata RHEA-2012-0356 did ensure on RHEL3/4/5/6 that the leap second inserted at the end of June, 2012 is properly recognized also on systems that are not connected via NTP.

RHEL 4
  1. There is a chance that the printing of this message can cause the kernel to crash in Red Hat Enterprise Linux 4; this issue is documented in System hangs on printing the leap second insertion message.
  2. Ensure that tzdata-2015a-1.el4 or later is installed so that this leap second may be inserted for systems not synchronized by ntpd; this package was released from RHEA-2015:0141-1.
RHEL 5
  1. There is a chance that the printing of this message can cause the kernel to crash in Red Hat Enterprise Linux 5; this issue is documented in System hangs on printing the leap second insertion message.
  2. Ensure that tzdata-2015a-1.el5 or later is installed so that this leap second may be inserted for systems not synchronized by ntpd; this package was released from RHEA-2015:0141-1.
RHEL 6
  1. There is a chance that a system can hang once it receives notification of the insertion of a leap second; this issue is documented in Systems hang due to leap-second livelock.
  2. After the leap second has been inserted futex heavy applications began consuming a large amount of CPU; this issue is documented in Why is there high CPU usage after inserting the leap second?.
  3. Ensure that tzdata-2015a-1.el6 or later is installed so that this leap second may be inserted for systems not synchronized by ntpd; this package was released from RHEA-2015:0141-1.
  4. The TAI offset is not updated correctly during the leap second; this issue is documented in TAI offset is incorrect during the leap second.
  5. Using -x with ntp still results in instantaneous clock changes when leap second occurs; this issue is documented in Does Red Hat plan to release xleap.patch with ntp?.
RHEL 7
  1. Ensure that tzdata-2015a-1.el7 or later is installed so that this leap second may be inserted for systems not synchronized by ntpd; this package was released from RHEA-2015:0141-1.
  2. Using -x with ntp still results in instantaneous clock changes when leap second occurs; this issue is documented in Does Red Hat plan to release xleap.patch with ntp?.

In addition to the issues tracked above it is possible that application-specific issues will arise if the leap second was not considered during development. Issues of this nature are documented in Libraries and Applications do not account for the Leap Second.

Note: Red Hat recommends that customers using PPC and IA64 architectures use the method described under Systems not running NTP or PTP.

Testing

Red Hat continues to test for any issues, and as information is available regarding the completed tests this article will be updated.

A lab has been developed to perform this test, and is available at Leap Second Vulnerability Detector.

In addition, Red Hat strongly recommends that customers test their builds and environments, and instructions on how to test, including a sample program to do so, are provided at Are we susceptible to a leap second event?.

Additional Resources

Additional information on leap seconds and how they are handled in Linux and by NTP can be found at the following links: