Resolve Leap Second Issues in Red Hat Enterprise Linux

Updated -

Contents

  1. Introduction
  2. Upcoming Leap Second Events
  3. Environment
  4. Understanding the Leap Second
  5. Handling of the Leap Second
  6. Known Issues
  7. Realtime Kernels
  8. Testing
  9. Example Configurations
  10. Additional Resources

Introduction

Leap seconds are a periodic one-second adjustment of Coordinated Universal Time(UTC) in order to keep a system's time of day close to the mean solar time. However, the Earth's rotation speed varies in response to climatic and geological events, and due to this, UTC leap seconds are irregularly spaced and unpredictable. This article seeks to provide information regarding the leap seconds themselves, along with addressing how Red Hat Enterprise Linux handles the insertion of one.

Upcoming Leap Second Events

At this time, we are not aware of upcoming leap second events.

Environment

  • Red Hat Enterprise Linux versions 4, 5, 6, 7 and 8

Understanding the Leap Second

The basic time for mostly all of the world's local time zones is called Coordinated Universal Time, UTC, which is derived from a bunch of atomic clocks which are distributed in several countries all over the world. The rotation of the earth is not very constant and varies a bit over time, while decreasing the mean rotation speed slowly. This is the reason why so called leap seconds are inserted into the UTC time scale, they adjust the UTC time to the real earth rotation.

Why this extra second? It exists because the rotation of the Earth on its axis, which determines the passing of days and nights, slows down over a long period, mainly as a consequence of Moon-Sun attraction effects. In addition, the Earth is affected by its internal (core, mantle) and external (atmosphere, oceans) constituents. Nowadays, though, time is measured largely by 250 atomic clocks belonging to several countries, which function by measuring the transition of energy levels in an atom. Together these clocks are used to calculate UTC, and as this time measuring mechanism is independent of the Earth periodically the two must be brought into alignment with a leap second. In addition, we have to consider that the length of the day is nowadays 2 ms longer than in the year 1820. Not surprisingly then, the Earth's rotation slowly gets out of synchronization with UTC.

The International Earth Rotation Service, IERS, is measuring the true earth rotation and determines when a leap second has to be inserted. Insertion of a leap second is always scheduled for the end of the last day of a month, preferably at the end of June or December, at UTC midnight. In the past all leap seconds had been inserted at either one of those times(*). Announcements whether a leap second is scheduled or not are published by the IERS in their Bulletin C. The current Bulletin C is published about half a year before the next possible date for a leap second.

The IERS Bulletin C #30 from July 2005 announced a leap second to be inserted at the end of December 31, 2005, at UTC midnight. This was the first leap second that had been inserted since the end of 1998. This is why many applications which had been developed during the previous 7 years could not handle the leap second correctly.

Since, the leap second was inserted at the same moment all over the world, the local (civil) time of the insertion depends on the local time offset from UTC, e.g. if the time zone is UTC +3h then the leap second will be inserted when the wall clocks show 3 hours after midnight.

The standard way to count UTC time across a leap second is:

  2011-12-31 23.59.57
  2011-12-31 23.59.58
  2011-12-31 23.59.59
  2011-12-31 23.59.60 <-- leap second
  2012-01-01 00.00.00
  2012-01-01 00.00.01
  2012-01-01 00.00.02

If the system clock is kept in TAI and a right/* timezone is used then 23:59:60 may be listed; however, as 23:59:60 does not exist in Unix's implementation of UTC then the linux kernel inserts the leap second by stepping the system clock back by one second on the first clock update after 0:00 UTC. In local time counting depends on the time zone offset, e.g. for UTC+3h you would observe the following:

2012-01-01  02:58:00.000
2012-01-01  02:58:00.500
2012-01-01  02:59:00.000
2012-01-01  02:59:00.500
2012-01-01  03:00:00.0xx <-- leap second inserted
2012-01-01  02:59:00.0xx
2012-01-01  02:59:00.500
2012-01-01  03:00:00.000
2012-01-01  03:00:00.500

Leap seconds are a discontinuity of civil time. The time does not continue to increase monotonically but it is stepped by one second. Let's have a look at the time stamps of a leap second which is inserted, and the second after that leap second:

  2011-12-31 23.59.60 <-- leap second
  2012-01-01 00.00.00

We can normalize the time and date of the leap second:
60 seconds are 1 minute, which lets the minutes increment from 59 to 60
60 minutes are 1 hour, which lets the hours increment from 23 to 24
24 hours are 1 day, which lets the date increment, and so on.
Finally we can say that both lines represent exactly the same time, or 2 consequent seconds have the same time stamp.

Several time dissemination services also propagate the announcement of a leap second after this has been determined by the IERS. This includes for example the German long wave transmitter DCF77 and the satellite based navigation system GPS, so receivers which decode the signals from those systems can also decode the leap second announcement. Applications which read the time from those receivers can also determine the leap second announcement if this information is included in the applied protocol, e.g the time string transmitted by the receiver.

Please note that time code receivers can only pass the announcement of a leap second to the application, and count the time correctly over that period. It's the task of the application and/or operating system to handle that leap second correctly.

(*): An additional leap second notification was erroneously issued by some NTP (Network Time Protocol) servers within the North America region for midnight UTC on August 31, 2012. Two stratum 1 servers -- truechimer.cites.illinois.edu and time-b.nist.gov -- were identified as erroneously issuing the leap second notification. Any higher stratum (stratum 2, 3, 4....) NTP server with either of these two servers in their hierarchy should be considered as having issued the leap second incorrectly. NTP clients using these servers will set the kernel's leap indicator flag, which cannot be cleared until the system is rebooted.

Handling of the Leap Second

The leap second will be handled in two primary ways, depending on how the system clock is configured.

The system clock is keeping time with UTC and synchronized via NTP or PTP

As mentioned previously, the leap second must be skipped when the system is using UTC, as this second does not exist. Therefore, to prevent the system clock from being one second ahead of UTC, the clock must be corrected after the leap second occurs. This correction may occur in a variety of ways, outlined below:

Notify the kernel, allowing it to step back the clock

The default approach by most daemons is to notify the kernel, and allow it to insert the leap second. For instance, during the last day before a leap second correction NTP servers should notify their clients that a leap second will occur, and at 23:59:59 UTC, the Linux kernel should add or remove an extra second by making the 60th second occur twice or removing it entirely. Thus, Red Hat Enterprise Linux systems running an NTP client, with a default configuration, during the last leap second correction should have counted time as follows:

2008-12-31 23:59:59:052549000 UTC    <-- 1st occurrence of the 60th second
2008-12-31 23:59:59:259988000 UTC
2008-12-31 23:59:59:465214000 UTC
2008-12-31 23:59:59:669629000 UTC
2008-12-31 23:59:59:873936000 UTC
2008-12-31 23:59:59:079184000 UTC    <-- 2nd occurrence of the 60th second
2008-12-31 23:59:59:284011000 UTC
2008-12-31 23:59:59:488648000 UTC
2008-12-31 23:59:59:692691000 UTC
2008-12-31 23:59:59:896577000 UTC
2009-01-01 00:00:00:052378000 UTC

When the leap second occurs, the kernel prints a message to the system log.

Have the daemon step back the clock

Instead of notifying the kernel, and allowing it to step the time back, the daemon may step the system clock backwards. Time will be counted similarly to the kernel inserting the leap second; however, as the daemon performs the actual insertion any bugs specific to the kernel will not be encountered.

Slew the leap second

RHEL 6.8 and RHEL 7.2 introduced an option to slew the leap second for systems using Chrony, allowing this second to be spread out over a period of time instead of an immediate step back in time. This slew is handled as a one-second offset accumulated at 00:00:00 UTC, for leap second insertions, or 23:59:59 UTC, for leap second deletions, that is smoothed out over time. This method may be preferred over stepping when applications running on the system are sensitive to jumps in system time and it is acceptable that the clock will be off for a longer time.

Ignore the leap second

Instead of addressing the leap second daemons may be configured to ignore it entirely, allowing time to be slowly corrected as any other unexpected change in time. The clock will continue with the incorrect time until it is addressed through normal operation. If multiple systems are configured with this method it is possible that their times will drift apart from each other, as the correction process is random to a certain extent. This method is not recommended for systems that require in-sync communication.

Leap smear on NTP Server

RHEL 6.8 and RHEL 7.2 introduced an option to smear the leap second when Chrony is running on the NTP server(s). Instead of having the clients slew the leap second, it may instead be suppressed on the server and slowly correct the served time by slewing the leap second instead of stepping it. Clients do not need any special configuration, as they do not know there is a leap second, and will continue to follow the server time until they are in sync with UTC. Smearing the leap second on the server is much slower than doing so on individual clients; however, it allows all clients to connect.

When configuring a NTP server to smear the leap second it is necessary to ensure that all clients using it are only pointing to servers that are performing the smear in an identical fashion. If clients are configured to receive updates from multiple servers, and these servers handle the leap second in a different fashion, inconsistent results will occur.

Table of Options

The following table highlights the options available to each daemon:

Method NTPD CHRONYD PTP4L PHC2SYS
Kernel Step Yes (default) Yes (default) Yes (default) Yes (default)
Daemon Step Yes(disable kernel with ntp 4.2.6) Yes (leapsecmode step) No No
Slew No Yes (leapsecmode slew) No No
Ignore Yes (-x) Yes (leapsecmode ignore) Yes (kernel_leap 0) Yes (-x)

A list of example configurations may be found at the bottom of this page, in Example Configurations.

The following table demonstrates the effects of each configuration method:

Client kernel step Client daemon step Client slew Client ignore (ntpd -x) Server slew (smear)
Continuous time No No Yes Yes Yes
Clock off for 1 second 1 second Seconds or minutes Hours Hours
Frequency error Zero* Zero* Large Small Small
Kernel bugs Yes No No No No
Clients stay synchronized to one another Yes* Yes* Yes No Yes

* Ignoring the discontinuity in time

The system clock is keeping time in TAI-10 seconds, is not synchronized, and is using a right/* timezone

Systems may also be configured to report time corrected for leap seconds by updating the tzdata package to the latest version available, copying the appropriate file from the /usr/share/zoneinfo/right directory hierarchy to /etc/localtime, and resetting the clock to the correct local time. The files in /usr/share/zoneinfo/right contain local time information corrected for all leap seconds that have occurred since the beginning of the Epoch on 1970-01-01 00:00:00 UTC. The other time zone files in /usr/share/zoneinfo do not have leap second corrections added. After the 2008 leap second, there were 24 leap seconds added since the Epoch. Any application that expects the time to be in UTC will have issues if a right/* timezone is used.

As an example, if a system is in the America/Los_Angeles (US Pacific) time zone, you can reconfigure the system to report leap-second-corrected time by running the following and resetting the clock to Pacific Time:

cp /usr/share/zoneinfo/right/America/Los_Angeles /etc/localtime

After /etc/localtime has been changed glibc will reload this file automatically; in addition, note that adjusting timezones does not change the system time, only the conversion from the system time to local time that happens in applications via glibc functions, such as localtime(), ctime(), etc. A system restart is not required after this update; however, if any applications are caching results from these functions then these applications may need to be restarted after tzdata has been updated.

Note: Red Hat does not recommend running without a source of time synchronization enabled as there is no guarantee that systems will remain in sync with each other. Time drift may occur on each system as determined by its clock source. The decision to run without NTP/PTP/Chrony will need to be made on a per-system or per-cluster basis with the understanding that time drift may occur.

By default, Linux systems not using NTP or PTP to synchronize their timekeeping will not correct for leap seconds, and the time reported by these systems will have a one-second difference relative to UTC after the leap second correction. You should reset the clock manually after leap seconds occur.

Known Issues

RHEL 4
  1. There is a chance that the printing of this message can cause the kernel to crash in Red Hat Enterprise Linux 4; this issue is documented in System hangs on printing the leap second insertion message.
  2. Slew mode does not prevent ntp from setting the kernel flag; this issue is documented in In Red Hat Enterprise Linux 4, slew mode does not prevent ntp from setting the kernel flag.
  3. For systems not synchronized by ntpd or ptp an updated tzdata package that contains the December 31st leap second is required. The updated tzdata package was released as part of RHEA-2016-1982, and any systems using RHEL 4 that are not synchronized by ntpd or ptp should update to tzdata-2016g-2.el4, or a later version, to receive this fix.
RHEL 5
  1. There is a chance that the printing of this message can cause the kernel to crash in Red Hat Enterprise Linux 5; this issue is documented in System hangs on printing the leap second insertion message.
  2. When running ntp in slew mode time is not adjusted properly by the leap second; this is documented in Time is not adjusted by a leap second when ntpd runs with -x option.
  3. For systems not synchronized by ntpd or ptp an updated tzdata package that contains the December 31st leap second is required. The updated tzdata package was released as part of RHEA-2016-1982, and any systems using RHEL 5 that are not synchronized by ntpd or ptp should update to tzdata-2016g-2.el5, or a later version, to receive this fix.
RHEL 6
  1. There is a chance that a system can hang once it receives notification of the insertion of a leap second; this issue is documented in Systems hang due to leap-second livelock.
  2. After the leap second has been inserted futex heavy applications began consuming a large amount of CPU; this issue is documented in Why is there high CPU usage after inserting the leap second?.
  3. The TAI offset is not updated correctly during the leap second; this issue is documented in TAI offset is incorrect during the leap second.
  4. Using -x with ntp still results in instantaneous clock changes when leap second occurs; this issue is documented in Does Red Hat plan to release xleap.patch with ntp?.
  5. Absolute timers may fire early when the leap second is inserted; this issue is documented in Absolute Timers that Expire at Midnight UTC May Fire Early When the Leap Second is Inserted.
  6. For systems not synchronized by ntpd or ptp an updated tzdata package that contains the December 31st leap second is required. The updated tzdata package was released as part of RHEA-2016-1982, and any systems using RHEL 6 that are not synchronized by ntpd or ptp should update to tzdata-2016g-2.el6, or a later version, to receive this fix.
  7. There is a chance that chronyd may crash when smearing the leap second; this issue is documented in Chronyd crashes when performing server leap smear.
  8. Some versions of NTP will continue to announce the leap second after it has been inserted; this issue is documented in The ntpd leap status is not reset after inserting a leap second.
RHEL 7
  1. Using -x with ntp still results in instantaneous clock changes when leap second occurs; this issue is documented in Does Red Hat plan to release xleap.patch with ntp?.
  2. Absolute timers may fire early when the leap second is inserted; this issue is documented in Absolute Timers that Expire at Midnight UTC May Fire Early When the Leap Second is Inserted.
  3. For systems not synchronized by ntpd or ptp an updated tzdata package that contains the December 31st leap second is required. The updated tzdata package was released as part of RHEA-2016-1982, and any systems using RHEL 7 that are not synchronized by ntpd or ptp should update to tzdata-2016g-2.el7, or a later version, to receive this fix.
  4. There is a chance that chronyd may crash when smearing the leap second; this issue is documented in Chronyd crashes when performing server leap smear.
  5. There is a chance that hrtimers may fire early when the leap second is inserted; this issue is documented in Hrtimers may expire early when a leap second is inserted.
  6. Some versions of NTP will continue to announce the leap second after it has been inserted; this issue is documented in The ntpd leap status is not reset after inserting a leap second.

In addition to the issues tracked above it is possible that application-specific issues will arise if the leap second was not considered during development. Issues of this nature are documented in Libraries and Applications do not account for the Leap Second.

Note: Red Hat recommends that customers using PPC and IA64 architectures use the method described under Systems not running NTP or PTP.

Realtime Kernels

A separate article has been created for tracking issues with realtime kernels. For more information refer to Resolve Leap Second Issues in Realtime (RT) kernels.

Testing

Red Hat continues to test for any issues, and as information is available regarding the completed tests this article will be updated.

A lab has been developed to perform this test, and is available at Leap Second Issue Detector.

In addition, Red Hat strongly recommends that customers test their builds and environments, and instructions on how to test, including a sample program to do so, are provided at Are we susceptible to a leap second event?.

Example Configurations

Stepping the leap second with Chrony

To step the leap second, add the following options to the /etc/chrony.conf:

leapsecmode step

These options will instruct chronyd to correct the system clock by stepping the leap second, bypassing any bugs with the kernel's insertion of the leap second.

Slewing the leap second with chrony

To slew the leap second, add the following options to the /etc/chrony.conf:

leapsecmode slew

These options will instruct chronyd to correct the system clock by slewing the leap second instead of stepping back immediately.

Smearing the leap second for clients on chrony NTP server

To smear the leap second, add the following options to the /etc/chrony.conf:

leapsecmode slew
maxslewrate 1000
smoothtime 400 0.001 leaponly

These options will instruct chronyd to correct the system clock by slewing the leap second and limit the slewing rate of the local clock to 1000 parts per million (ppm).

Ignore the leap second with NTPD

To ignore the leap second, allowing it to be corrected by normal slewing operations, ensure the following line is present in /etc/sysconfig/ntpd:

OPTIONS="-g -x"

With this flag added the kernel's leap second correction is disabled, and the leap second will be corrected over a period of time.

Additional Resources

Additional information on leap seconds and how they are handled in Linux and by NTP can be found at the following links:

54 Comments

"There is a chance that the printing of this message can cause the kernel to crash in Red Hat Enterprise Linux 4 and 5"

What is the issue in kernels prior to version kernel-2.6.18-164.el5 that may trigger kernel crash?

Was tracked in Red Hat Bugzilla - Bug 479765 - Leap second message can hang the kernel

java died today because of leap second. my rhev manager was using all cpu. had to restart the machine. restarting jbossas was not the answer.

All of my tomcat6-on-RHEL6 servers froze. You could still ssh in, tomcat will still give some responses, but a server that could normally handle 1000 req/s was unable to handle 100 req/s until after a reboot.

All of my RHEL6 machines are pegged at 100% CPU and very high load. Restarting services doesn't help and disabling NTP doesn't do anything. Splunk is freaking out and causing issues for me. All of the RHEL5 boxes are fine. I ran "uptime" against my RHEL6 boxes using mcollective, and all the servers are at a load of 5 or higher. What a mess.

Also, my monitoring systems show that all of this started at 5PM PST (12AM UTC) exactly. I see a 100% spike in CPU and load at exactly that time. So I'm pretty certain this is related.

ok so this fixed it for me without a reboot. CPU dropped and everything's going back to normal after running this simple command: /etc/init.d/ntpd stop; date; date `date +"%m%d%H%M%C%y.%S"`; date

Thanks a ton Pedro Alves! http://pedroalves-bi.blogspot.com/2012/07/java-leap-second-bug-how-to-fix-your.html

That really helped on RH6 machine with java. Load average dropped to normal.

Thanks Pedro Alves and DTA

More here:

http://serverfault.com/questions/403732/anyone-else-experiencing-high-rates-of-linux-server-crashes-during-a-leap-second

I had the same issue on RHEL5 and 6 running Jboss

kernel versions
2.6.32-220.7.1.el6
2.6.18-308.1.1.el5

date -s "`date`"

did fix it for me

Its not just JBoss, I have used the above date command to calm 4 thrashing JVMs, none of which were running JBoss.

2.6.32-220.23.1.el6.x86_64

Does anyone know if there are other side effects of this issue and whether the date commands listed above will fix them too?

Thanks

Red Hat is aware of the upcoming June 30th, 2015 leap second and is working to address it.

"I have updated the RHEL server using steps "Systems not running NTP or PTP".
Manually updated the latest tzdata rpm to "tzdata-2015a-1.el5.i386.rpm" . But I am not sure how to test this fix as it working fine.

root# rpm -qa | grep tzdat

tzdata-2015a-1.el5

root# ls -l /etc/localtime

lrwxrwxrwx 1 root root 33 Jun 30 2015 /etc/localtime -> /usr/share/zoneinfo/right/PST8PDT

root# ntpq -c "lassoc" -c "mrv &1 &999 leap,srcadr,stratum"

ind assID status conf reach auth condition last_event cnt

1 20766 9614 yes yes none sys.peer reachable 1
assID=20766 status=9614 reach, conf, sel_sys.peer, 1 event, event_reach,
srcadr=LOCAL(0), srcport=123, dstadr=127.0.0.1, dstport=123, leap=00,
stratum=10, precision=-20, rootdelay=0.000, rootdispersion=10.000,
refid=LOCL, reach=017, unreach=0, hmode=3, pmode=4, hpoll=6, ppoll=10,
flash=00 ok, keyid=0, ttl=0, offset=0.000, delay=0.000,
dispersion=938.151, jitter=0.001,
reftime=d93b89d0.3e6101d4 Mon, Jun 29 2015 2:09:36.243,
org=d93b89d0.3e6101d4 Mon, Jun 29 2015 2:09:36.243,
rec=d93b89d0.3e611681 Mon, Jun 29 2015 2:09:36.243,
xmt=d93b89d0.3e60efb8 Mon, Jun 29 2015 2:09:36.243,
filtdelay= 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00,
filtoffset= 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00,
filtdisp= 0.00 0.95 1.89 2.84 16000.0 16000.0 16000.0 16000.0

The leap still shows as leap==00. I have changed the dates manually as well but nothing helps.

Any help or suggestion that how to confirm that the upgrading to the latest rpm and modifying the localetime link will fix the leap second issue."

Hi, I guess, there was misunderstanding.
First of all you are not using ntp so you will not see that whether ntp has aligned to issue a leap second or not(locally) using ntpq command.

You can verify it using zdump command to see if system will feel leap second or not. For me @IST.

zdump -v right/Asia/Kolkata

right/Asia/Kolkata Tue Jun 30 23:59:60 2015 UTC = Wed Jul 1 05:29:60 2015 IST isdst=0 gmtoff=19800 << Look at time
right/Asia/Kolkata Wed Jul 1 00:00:00 2015 UTC = Wed Jul 1 05:30:00 2015 IST isdst=0 gmtoff=19800 << Look at time

What about issue in RHBA-2015:0690-1? This is not listed in issue of RHEL 6?

Hello Sam,
we will be updating this article regarding ntp slew mode or RHBA-2015:0690-1

Is system reboot necessary after updating only tzdata package?

I have this same suspicion. As my /etc/localtime is not a symbolic link to any /usr/share/zoneinfo/*. how to ensure we´re ok?

Latest tzdata RPM "tzdata-2015a-1.el5" is installed.
$ rpm -qa | grep tzdata
tzdata-2015a-1.el5

And for PST8PDT timezone leapsecond is falling on 30th June.
$ zdump -v right/PST8PDT | grep 2015
right/PST8PDT Sun Mar 8 09:59:59 2015 UTC = Sun Mar 8 01:59:59 2015 PST isdst=0 gmtoff=-28800
right/PST8PDT Sun Mar 8 10:00:00 2015 UTC = Sun Mar 8 03:00:00 2015 PDT isdst=1 gmtoff=-25200
right/PST8PDT Tue Jun 30 23:59:60 2015 UTC = Tue Jun 30 16:59:60 2015 PDT isdst=1 gmtoff=-25200
right/PST8PDT Wed Jul 1 00:00:00 2015 UTC = Tue Jun 30 17:00:00 2015 PDT isdst=1 gmtoff=-25200
right/PST8PDT Sun Nov 1 08:59:59 2015 UTC = Sun Nov 1 01:59:59 2015 PDT isdst=1 gmtoff=-25200
right/PST8PDT Sun Nov 1 09:00:00 2015 UTC = Sun Nov 1 01:00:00 2015 PST isdst=0 gmtoff=-28800

Now to check the vulnerability used the " Leap Second Vulnerability Detector" tool and the result says "Not vulnerable".
$ ./leap_vulnerability.sh
Installed kernel version: 2.6.18-371.1.2.el5PAE
Installed tzdata version: tzdata-2015a-1.el5
Not vulnerable

Now we are confused as zdump output says leapsecond is coming and the "Vulnerability Detector" tools outputs says we are safe.

Hi,

Output is expected.

Detector tool just check whether we have latest packages updated or not (which handle leap second properly). If it detect package which do not handle it properly, it will show it is vulnerable.

"Output says we are safe" means system will handle leap second properly.

For more information read this article Resolve Leap Second Issues in Red Hat Enterprise Linux.

Any idea when we can expect the updated article covering ntp slew mode?

all of my servers have the ntp client running. From my understanding if ntp is running, I should not worry about kernel crash even though the leap_vulnerability.sh says to update the tzdata package?

As long as the kernel is updated you should not have to worry about a crash; the tzdata package is primarily for when the right/* timezones are in use. If a timezone from the right/* directory is not in use then you will not require the update to the tzdata package.

Even though I am not using timezone from the right/* directory, It still doesn't hurt to update the tzdata package?

It won't cause any issues (and we typically recommend running the latest packages if possible), but it will not be necessary for the leap second if you're not using a timezone from the right/* directory.

Hi, if we update the tzdata RPM and our Systems not running NTP or PTP,should we update the kernel to pervent the knowen issues above in the article,like kernel crash?

Hey. No worries about rebooting a hundred servers / updating. Plan B is faster. Just /etc/init.d/ntpd stop an hour before the leap and get it back on after the event. That's all.

Cheers!!

In that case, could you help me understand how the leapsecond-flag in the kernel would get cleared? With the flag set, the kernel is handling the leapsecond, from what I see.

Hi Christian. Is not big deal. Already talk to RedHat support. Forget about the flag. You turn off ntp in order to avoid a possible brutal desynchronization (+leap second) that some kernels will not manage. You turn it on a few hours later and that's it.

Hi All,
In our environment we are synchronized with a 3rd party Vendor NTP server using Solaris 9 NTP server.All our client server are RHEL 4/5/6 and is synced to the Solaris server. And as we confirmed with the 3rd party vendor they will insert a leap second at June 30th, 2015 at 23:59:60 UTC.
Can someone help to clarify below concerns.
1. To avoid any impact to all clients is it safe to run the NTP server within our premise in slew mode ?
2. If we enable slew mode in server side does all client also need to be changed to slew mode ? ( does change in NTP server side alone fix ?
3. As confirmed with Redhat in a separate case , if the kernel versions are updated to the recommended level then we shouldn't hit the OS crash issue when leap second insertion occur. However High CPU can occur for some applications irrespective of kernel version or ntp package version , Can slew mode can be used to avoid this situation ?
4. Any other tips to better handle this situation would be much appreciated.

Regards
Ajay.

Hi all,
can happen that cron scripts scheduled on 0:00 would be launched twice?

Thank you very much!

Kind regards

Jaroslav

According to RH support tests the cron jobs scheduled on 0:00 will not be affected and should run just once as expected.

I am under the impression from the redhat article that if running ntp client that I should not worry about a kernel crash. I have some servers that need a kernel update however my oracle DBA's do not want me to update the kernel because their ASM disks /apps will no longer work with a new version, they would need to rebuild and tweak their apps. But what I am reading in the posts is that if the kernel is not updated to the latest version, then when the leap second happens, this could cause a kernel crash. .stuck between a rock and a hard place

for the leap second my servers are running ntp client. the tzdata is up to date, but there are a couple of kernels that need to be updated but our oracle DBA's do not want the kernel updated. Do i need to stop ntp and restart after the leap second happens or run ntp in skew mode, or just keep ntp running and hope that no kernel crash happens.

Dear All,

unfortunately our satellite channels have multiple 2015 released tzdata version are there..

tzdata 2015a-1.el5
tzdata 2015b-1.el5
tzdata 2015c-1.el5
tzdata 2015d-1.el5

when I run yum list tzdata, I got only tzdata 2015d-1.el5 available for update but in package change log this one showing related to Egyptian DST stuff, but when i check the same for tzdata 2015a the one showing leap second issue.

Now guys could someone correct do we need to go for 2015a or 2015d version and how to get only the version related to leap second issue (may be 2015a)?

Does anyone ran into similar kind of issue ?

Thanks

Hi Kishore,

use the latest tzdata available (2015d in this case). Newer tzdata include changes which arrived in previous versions (if those are still valid, of course...).

Kind regards

Jaroslav

Thank you Jaroslav, that make sense.

Actually Redhat mentioned 2015a package across all leap second articles so worried after looking at change log on that particular packages.

Is there is new tzdata package for RHEL4 or is the latest tzdata for v4
rpm -qa |grep tzdata
tzdata-2011n-2.el4

Larry,

For RHEL4, the latest ntp tzdata package is tzdata-2015d-1.el4. The latest package is shipped via RHEL 4 Extended life cycle channel. To access the channel, you need to have Extended Lifecycle support subscription

Thank you Ranjith for your response. We do have the Extended Life Cycle support subscription, I updated the base channels. however we have 4.7 and the channel that i had to choose from was 4.8 and i did not see any errata package for tzdata. Am i looking in the wrong place. Also we have some kernels that should be updated, however Oracle DBA's do not want to update because it will mess with their ASM configurations/applications. I have ntp client running. Should i be worried about a kernel crash?

Hello Larry

If you have ELS, then you should be able to access the below RHEL 4 ELS tzdata package url

https://access.redhat.com/downloads/content/rhel---4/x86_64/2996/tzdata/2015d-1.el4/noarch/db42a60e/package

Contact Red Hat Technical Support if you need assistance for subscribing the system to the appropriate ELS channels

Regarding kernel update

As mentioned in this article, For RHEL4 make sure kernel is above kernel-2.6.9-89.EL

If you are not able to update the kernel due to application dependency, then disable ntpd service on the system prior to the event (minimum 24-48 hours) and then start it post the leap second event.

Some NTP servers can notify the ntp clients about leap second even prior to 24/48 hours. using the command ntpq -c "lassoc" -c "mrv &1 &999 leap,srcadr,stratum" name1.test.com | grep -E "leap=([^0].|.[^0])" you can check whether the ntp client has already received the leap second notification. If you notice leap=01 in the above command output, then it has received the leap second event.

If the NTP client has received the notification, then you cannot clear the leapevent from the kernel side. You will have to reboot the node. Prior to reboot disable ntpd service from autostarting (chkconfig ntpd off) and then after reboot chkconfig ntpd on; followed by service ntpd start

So it would be better to verify whether the RHEL4 received the leap second notification if not then configure ntp service in slew mode

If you have any further questions, I would recommend to contact Technical support so that we can review our environment and answer your questions better

Correction:

If the NTP client has received the notification, then you cannot clear the leapevent from the kernel side. You will have to reboot the node. Prior to reboot disable ntpd service from autostarting (chkconfig ntpd off) and then after reboot chkconfig ntpd on, Configure NTP in slew mode and then start ntpd service

Ranjith,

Thank you for this information. It is very helpful.

"Systems running any version of Red Hat Enterprise Linux should automatically account for leap second corrections if they are using the NTP (Network Time Protocol) daemon to synchronize their local timekeeping with an NTP server."

This is not technically correct. If the NTP server that you point to does not have a leap file with leap second data accounted for, then the NTP client will not get the leap second. You will need to check for the leap second flag on the client if you do not own the NTP server.

Dear Friends,

I got below Workaround from Redhat. No kernal upgrade, no tzupdate. No reboot server.

Red Hat strongly recommends that customers update their systems to the most recent available Red Hat Enterprise kernels.

If it is not possible and you want to avoid the Leap Second insertion, you should disable ntpd daemon by running "service ntpd stop" before Jun 30th.

At some point in the future you might want to sync the systems again with the rest of the world.
These methods could be used:

  • stepping the clock: when running ntpd again ("service ntpd start"), the clock will be stepped
  • correct the clock slowly: ntpdate -B could be used, the correction takes about 2000 seconds. After that, ntpd can be started again and the error will be already corrected, so ntpd won't need to perform any steps.

Hello Chunhui,

yes, you can go for workaround listed here. https://access.redhat.com/articles/199563.
Note1:- On RHEL 5, stopping ntp must be done before last 24 hours. Once kernel receive that informations, it can't reset.

In case of NTP is not running , if the TZData is up-to-date , is that will automatically correct the kernel to insert the lead second

Please have a look at the Leap Second queries related to tzdata article.

Abdul, in order for tzdata to correct for the leap second you have to copy over or soft link to the right/* timezones to /etc/localtime (rpm -ql tzdata | grep right).

But generally speaking we don't recommend that you do that (as noted above). As with a disabled ntp your systems can stray away from each other at different rates. So you'd probably be better off just using slew/-x option or just go with the default ntp configuration that will step.

The major issues last time around were do to hangs, these have all been fixed with kernel updates.

Usage of ntp configured with -x/slew will make your systems take longer to time correct when not going outside of the step threshold (600s)

Default configuration of ntp (without -x) will cause your system to step, but it should time correct much quicker than a typical slew correction.

We've documented two best approaches to workarounds for leap second insertion concerns.

How to clear the Leap Second Insertion flag after it has been received?
https://access.redhat.com/articles/199563

Now if you've been able to update your kernels, you're most likely going to be fine with ntp running in step (without -x) mode or in slew (-x) mode without having to worry about those two workarounds noted above.

Hope this helps, let us know if something isn't clear.

Hi All,
Am I right saying that after the kernel is up-to-date being not vulnerable the server either needs to have a tzdata installed newer than tzdata-2015a-1.el5 if NTP is not running. I understand this does not need a system restart but is ntpd needs to be restarted?
Also in case of using NTP, can the leapsecond notification passed over multiple hops? Meaning that if my system's NTP server(not having up-to-date tzdata) but pointing to the RedHat pool then everything should be fine? Thanks

@Dave Sullivan--
Aren't the right/* timezones for TAI? This would cause the time to jump ahead by 25 seconds?
This is from https://access.redhat.com/solutions/1462333

Hi All,

If i have a system running at a kernel level which could hang once it receives notification of the insertion of a leap second and i'm running the latest version of tzdata that handles the leap second what could happen? I believe the system could still hang if it's on a unpatched kernel as above?
Thanks

Please have a look at the Leap Second queries related to tzdata article.

Pages