Kernel Developers Tracking Down New Year's Eve Leap Second Issue

The Linux admins who experienced the crashes started a thread to that effect on the slashdot.org community site. In response, developer Linas Vepstas pulled together a summary of the issue on the Kernel mailing list lkml.org.

According to Vepstas, the 53 reported hard crashes at or near midnight December 31 2008 had a few things in common. In all cases the systems were not pingable and power-off cold reboots were required to get them back to normal working conditions. There were no syslog error messages, no kernel oopses and no core dumps.

The likely suspect was the worldwide coordinated leap second required to get atomic clocks back in synch with the earth's orbit. It turns out the last minute of 2008 was actually 61 seconds long. Even the New York Times had forewarned about it.

Network Time Protocol (NTP) clocks needed to bump their times by this second, the first time since 2005. Vepstas suspects that this jump prompted a race condition in the kernel that led to the Linux crashes.

The lkml.org mailing list is currently discussing if and how the crash condition can be reproduced, how Linux can deal with it in the future and what effect leap seconds will have on the UNIX time calculation.