Bruised by past mistakes, tech firms brace for 'leap second'

The leap second, as it's called, is needed to keep UTC in line with solar time. The two get out of whack due to changes in the earth's rotation, and 25 leap seconds have been added to clocks since 1971. Network Time Protocol (NTP) helps regulate the official time among Internet servers, keeping it in sync with UTC.

But the last leap second in 2012 took some IT companies and other firms by surprise, and caused websites including LinkedIn and Reddit, as well as Qantas' passenger reservation system, to crash. The problems involved unpatched Linux OS kernels, Hadoop instances, Cassandra and MySQL databases and Java-based programs.

Linux systems in particular were the focus of discussion after the 2012 leap second, as the bug caused everything from slowdowns in performance to overactive systems that led to CPU locks, Ron Pacheco, director of product management at Red Hat's Platforms Business Unit, said via email.

This time around, however, vendors say they are better prepared.

"Our small-scale tests are promising, and we'll be watching during the event to quickly respond to any unforeseen issues that may arise," a spokeswoman for Reddit said via email.

A spokeswoman for LinkedIn, meanwhile, said clock adjustments are being made to prevent any problems.

Qantas, one of the first major companies affected by the leap second in 2012, was hit by computer outages that delayed flights due to the effect of the Linux bug on the Amadeus reservation system, produced by Spain's Amadeus IT Group. The system is used by dozens of airlines around the world.

"We have sought and received assurances from Amadeus that they have taken action to make sure that the same problem does not happen again this year, and we're confident that it won't," a Qantas spokeswoman said via email. Amadeus did not respond to requests for more information.

There are several methods of dealing with the leap second. Google, for instance, implements a "smear window" centered on the leap second. To ensure that its NTP servers are in sync with the extra second, they are slowed, or "smeared," by about 14 parts per million.

"Twenty hours later, the entire leap second has been added and we are back in sync with non-smeared time," Google engineers Noah Maxwell and Michael Rothwell wrote in a blog post last month.

Another approach is to have servers simply count the 60th second twice at 23:59:59 UTC, Red Hat's Pacheco said.

The leap second will kick in at 9 a.m. Wednesday in Japan, just as businesses start work. Dominant mobile carrier NTT DoCoMo has programmed its servers to slightly extend the length of each second over a couple of hours before 9 a.m. to stay in sync with UTC, a spokeswoman said. Rival SoftBank and major messaging app Line said they are also taking countermeasures.

"We may see some small incidents with in-house computer systems or ones that are very old and not well maintained," said Tetsutaro Uehara, a computer science professor at Ritsumeikan University in Kyoto. "But they won't cause big problems like we saw in 2012."

With the concern it has caused among IT companies as well as stock market regulators, the leap second has earned its share of detractors. Representatives of various countries will continue to debate whether it should be abolished at a November meeting of the International Telecommunication Union (ITU) World Radiocommunication Conference. One alternative is a continuous time scale without leap seconds that could be based on UTC.

"The suppression of the leap second would facilitate a continuous time scale that would support all modern electronic navigation and computerized systems and eliminate the need for specialized ad hoc time systems," an ITU spokesman said via email.

Tim Hornyak covers Japan and emerging technologies for The IDG News Service. Follow Tim on Twitter at @robotopia.