'Daylight Savings Bugs' Loom

An anonymous reader writes "ZDNet has front page coverage of the looming daylight savings changeover, and the bugs that may crop up this year. With the extension of daylight savings time by four weeks, some engineers and programmers are warning that unprepared companies will experience serious problems in March. While companies like Microsoft have already patched their software, Gartner is warning that bugs in the travel and banking sectors could have unforeseen consequences in the coming months. ' In addition, trading applications might execute purchases and sales at the wrong time, and cell phone-billing software could charge peak rates at off-peak hours. On top of that, the effect is expected to be felt around the world: Canada and Bermuda are conforming to the U.S.-mandated change, and time zone shifts have happened in other locales as well.'" Is this just more Y2K doomsaying, or do you think there's a serious problem here?

Things you should know.

http://www.bloomingtonlinux.org/wiki/DST_Time_Chan ge_Issues [bloomingtonlinux.org]

A year ago, after most of Indiana went through its first timezone change in 40+ years, we found out that it presented a few problems in Linux, I tried to post a story to Slashdot about it to warn other people in the US that they would be dealing with this problem later when the rest of the US changes to the new DST. I tried several times to post it and they were all rejected.

Basically, you need to make sure that if you change your timezone data on your system that you restart everything, otherwise when the time does change, some programs continue to use the old timezone data and are an hour off.

Re:Things you should know.

A year ago, after most of Indiana went through its first timezone change in 40+ years, we found out that it presented a few problems in Linux, I tried to post a story to Slashdot about it to warn other people in the US that they would be dealing with this problem later when the rest of the US changes to the new DST. I tried several times to post it and they were all rejected.

Mod parent down, informative.

Re:Things you should know.

Our datacentre has ~ 500 Solaris / HP-UX / AIX boxen, and ~ 1000 Windows servers.

15 minute change window to apply patch, another 15 minutes to reboot successfully and come back online. Multiply 30 min x 1500 = 45,000 minutes, or 750 hours. But we only have one weekly change window, Sunday mornings from 2-6am. Assuming finite number of staff, contingency (there's always going to be some problems), etc... we started last September. We might just make the deadline.

So yes, I think its a bit of a problem. There's also the unspoken assumption that people learned their lessons during Y2k and have sufficient date handling logic to address changes to DST...nothing hard coded in the underlying applications.

Re:Things you should know.

Multiply 30 min x 1500 = 45,000 minutes, or 750 hours
Yeah that sounds about right if you do each machine one at a time. I should hope a datacentre is a little more efficient than that.

Re:Things you should know.

> 15 minute change window to apply patch, another 15 minutes to reboot successfully and come back
> online. Multiply 30 min x 1500 = 45,000 minutes, or 750 hours

Yeah, someone needs to tell my local cinema about that. They only show films for a few weeks, but with 1000 people in each showing, and the film lasting 90 minutes, that's 90,000 minutes, or nearly 9 weeks! There's going to be a lot of disappointed people...

Re:Things you should know.

That's a damned lie and demeaning to Windows to single it out like that.

MS-DOS is the same way. Apple ProDOS too.

Re:Things you should know.

Most programs that use the standard C library do use UTC and just don't realise it. The most important thing to realise about daylight savings time is that Time isn't changing. The sun still rise the after DST as before DST (astronomical adjustments due to Earth's heliocentric orbit not withstanding). But, how we read the clock is changing.

I heard one company just say, "we're going to just change the clock on the computer." This makes me cringe.

All file time timestamps on all versions of UNIX and versions of Windows derived from NT store times as Julian seconds since 1970-01-01 00:00:00 GMT. Changing the system clock means that the times will be stored wrong even though they display properly in the local timezone.

Other places to watch out for are applications that manage their own timezone data. Java is a prime example. Major database vendors would be another.

Re:Things you should know.

When you are so careful about file timestamps, you should know that Windows does not translate them correctly anyway.
Windows has no knowledge about timezone history. It translates the UTC time to local time using its current time-offset, which depends on the current DST status only.
So, when you now look at a file timestamp (in the GUI) that you created last summer, you will find that its time is one hour off.

Even when Microsoft would finally fix this (they consider it a feature rather than a bug), they would probably not fix the historic aspect.
I.e. now that the beginning of DST shifts one month, and you would look at a file created last year in that one month window, it would probably still be off.

Timezone handling in Windows just sucks. It does not have to be that bad, it works fine in Linux (including historic changes). Microsoft just has decided to make a bad implementation and then never fix it, in the name of backward compatability.

Re:Things you should know.

Why do your programs use the local timezone, anyway? Programs should handle and store dates in UTC, and convert to the local timezone only for display.

Saving the time in UTC does not get rid of all problems. Programs that store the data for future events will still run into issues. An event that was stored in the database before the system was patched will be off by one hour if the time falls within the dates that have the daylight savings rule changed.

Re:no need o worry

daylight savings times and zones change constantly in australia and everything is fine here

Everything is fine in Australia? Remember folks, this announcement is coming from the country that gave us The Wiggles.

y2k = media working for once

The reason y2k was such a letdown was because everybody took heed of the media hype and patched their stuff. If there had been no hype before, there might have been the problems that the hype was warning about. (or not, sensationalism is sensationalism)

Its like you're driving along and there's a huge backup for miles because of an accident in the middle of the road after a bend. Now this huge backup may have slowed you down and made you aware that there was a problem. If it was just you and the wreck, you may have plowed into it if you weren't paying attention.

Same thing with this hype. We should tolerate the hype because the heightened coverage will get bosses talking to programmers about fixing the software, and it'll turn out to be nothing.

Re:Mod Parent Up

Where I worked, we had a lot of old but reliable systems. We tested our mainframes, minicomputers, PCs, and network systems - they failed in varying and catastrophic ways.

We spent millions of dollars for all new systems, and thousands of hours in planning, procurement, implementation, and testing. We literally pushed all the boxes on the datacenter floor to the wall and built anew.

It was a horrendous chore, and I didn't get to spend New Year's with my family.

Perhaps we should have let you all freeze in the dark.

Re:y2k = media working for once

I think y2k was a let down because it was fairly straightforward to fix software and tell if it was going to get bitten or not. Basically anything that stored dates as two digits had to be fixed. The bigger problem will come in 2037 when lots of clunking software with no source code wraps around.

The DST thing is pretty evil too because it's usually up to runtime stacks like Java and CRT to decide on the timezone and time. If they give you the wrong time you're screwed. For the most part you might be okay if everything resolves down to some registry entries or timezone data files but that isn't always the case. There are functions such as Microsoft's _tzset() which are HARDCODED to a particular behaviour and apps that link to the CRT or have their own DLLs will be broken unless you recompile them.

Re:y2k = media working for once

Mod parent up!

Changing daylight savings time or y2k will be childsplay compared to the Year 2038 32-bit time_t overflow [wikipedia.org]. That is a really big problem with no easy fix. 32-bit times/dates are in everything from VCRs and microwaves to servers and desktops. 2038 will be everything that Y2K wasn't.

In principle, Linux and friends can fix this by redefining time_t to 64-bit - but lots of communication protocols and even file formats like tar use 32-bit dates. Admittedly we have 30 years to fix it, but we will need all 30.

Re:Y2K was an oddity and mis-explained

"In no way were dates routinely stored as two byte characters (99 being the max) when 1 byte would get you to 255 easily." Wrong. In EVERY WAY dates were routinely stored as three sets of two byte characters.

What you are completely ignoring is that the vast majority of the code that had to be examined and patched was written in COBOL. COBOL that store dates as a string of six digits. Digits that were stored in many cases as EBCDIC characters, not hexadecimal integer values. And just to make it fun, in some cases the source code was not available.

"[A]nyone that created a four digit date by String Concat: "19" + String(date) would " probably not have been born yet when the programs that needed to be fixed were written. It wasn't the programs that were written in the 1990's that had to be dealt with, it was the ones written in the 1960's. And if you don't believe there were any of those in use, then I suggest you have no idea what's really happening at your bank. Or in the US air traffic control system, for that matter.

rates?

and cell phone-billing software could charge peak rates at off-peak hours

Aiyeeee!!!!!!!!!!!!!

Re:rates?

And could charge off-peak rates for peak hours. The number of hours for peak and non-peak will remain the same, only the start times will change.

Everyone wants a credit when they are over billed, but no offers to give money back if they are under billed.

Re:rates?

Except that peak and off-peak mean something, namely volume of calls. So as a matter of fact, no, they are not "equally likely to charge you off-peak rates during peak periods" because you're more likely to be making a peak period call than an off-peak call.

A site that will give you USEFUL Info.....

... On how to deal with this is below:

http://www.reganfamily.ca/dst/ [reganfamily.ca]

This is likely more useful than the original article. It has resources for everything from Blackberries to UNIX.

Not such a big deal

Is it a problem? Yes, but it's nowhere near as big an issue as Y2K was. The biggest issue for my company is that many of our machines are 2000, so we had to create our own patch, since Microsoft is only patching 2000 for people who pay their extortion fees.

The majority of our applications just go off of the OS time, so as long as the OS is patched, everything else is fine. The DBA's will be coming in over a weekend to test the patches on the Unix servers, and the Server guys will be doing the same for the Windows servers, but other than that, there's not that much we have to do.

The financial industry will probably have more problems than most, but still, it should be negligable compared to Y2K.

Re:Not such a big deal

We have literally hundreds of servers running Windows 2000, and this DST patch was a major headache. As the parent noted, Microsoft did not include Win2K in their publicly released update.

There is a freeware utility to apply the DST patch on Win2K machines here [intelliadmin.com] (as a bonus, it also supports WinNT).

Note that you may also need to update the Java JRE/JDK.

Ahem, Not Exactly

While companies like Microsoft have already patched their software,

Ahem, not exactly. No patch for the perfectly good Exchange 5.5 server we're using with Outlook 2000. Suddenly we have to update to the latest Exchange and Outlook 2003 on every d@mn desktop. And I'm in Arizona were we don't even have daylight savings time!!!

Cool

And I thought that the year 1906 would pass with out any issues.

Doomsaying?

I don't see anyone saying planes are going to fall out of the sky or anything like that. Trades could be executed at the wrong time, costing people money. Cell phones could charge peak rates at the wrong times, costing people money. These could very easily happen if these companies were not on the ball about getting this patched early. Luckily, most operating systems required a simple patch (sometimes a reboot) to fix this, and that's about it. The extensive code fixes that the Y2K bug required simply aren't necessary here.

However, because of the perceived simplicity of the fix, there is a real possibility that some companies waited too late to address the issue and may not make it in time. This may cause minor glitches, but it's not like the nukes are going to start flying.

As for Y2K, obviously the people who were stockpiling ammunition and moving to the mountains were nuts, but there were real problems that could have occurred that did not because of the countless hours that were put in to fix the issues. It drives me crazy that after we spent millions of dollars and countless man hours fixing buggy code for Y2K, people look back and see that nothing happened and think all that money was a waste. If all that effort had not been expended, more computer systems would have had problems, and so the money was definitely not wasted. During Y2K, there were scattered reports of various computer systems crashing. It is likely there would have been many more such reports had we not taken the steps we did to address the issue.

Bermuda?

I don't see why Bermuda bothers with DLST. They are close enough to the equator and isolated enough that staying on normal time year round shouldn't interfere with commerce in any significant way.

Already has ramifications

Newer JDK's already have the new time zone rules encoded, and this can cause subtle bugs to suddenly surface. It turns out that Date.equals() does a deep object compare, including the time zone rules (not just which time zone you're in, but the rules regarding when daylight savings starts and ends). Thus if you have multiple JVMs involved, such as one on a database server and one on an application server running slightly different JVM revs (e.g. 1.4.2_08 vs. 1.4.2_11), then naive date comparisons (using equals() instead of equality on getTime()) can fail.

Get rid of daylight saving altogether

It serves no useful purpose any longer in what is almost a 24 hour society. What difference does it make to the vast majority of people what time the sun rises and sets anymore? All it does is add a small extra layer of confusion and complexity thats no longer necessary? If people really don't want to get up when its dark then go to work an hour later and leave an hour later. With flexitime its really not an issue anymore.

Herding cats

Let's switch to the metric system while we're at it.

Old OSes and Old JREs are the biggest concern

Think of how many companies have old systems that just continue to run forever. Most OS vendors drop OS patch support after about 5 years.

Okay, so all system processes should use UTC. We all know that. Users don't set their watches to UTC though.

Want a DST patch for Solaris 8? RHAS 2.1? Windows NT? You're going to have to shuffle and maybe you'll need to update the timezone files with 'zic' yourself. Have hundreds or thousands of these machines. Sucks to be you.

Oh, and the big killer is that Java has timezone rules embedded in it. That's right. Java VIRTUAL MACHINE. Java tracks timezones and DST changes INDEPENDENT of the OS since Java wants to be it's own OS.

So, if your company standardized on j2ee when you moved off the legacy systems for y2k, I'll almost bet you that the OS those java apps are running on won't have DST patches from the vendor, and your apps could have multiple JVMs that contain the wrong DST rules. You'll need to fix both of those if your java apps have anything to do with timezones and if you care about the times displayed.

I'd really like to get a list of everyone who voted for the 2005 dst timezone changes and start a movement to make them take responsibility for the huge business cost of their stupid legislation. It has to be 100X the cost of what they expected the changes to save...

A round of applause for the tz guys

The tz database http://www.twinsun.com/tz/tz-link.htm [twinsun.com] underlies time zone handling for the GNU C Library, FreeBSD, NetBSD, OpenBSD, Mac OS X, Solaris and many more, and is kept current by a dedicated team of (mostly?) volunteers. For time nerds, the historical comments in the plain text files of the tz ftp distribution (ftp://elsie.nci.nih.gov/pub/tzdata2007b.tar.gz [nih.gov]) are required reading.

If you're a Firefox person, FoxClocks (see my URL above) puts nice little world clocks on your statusbar. And yes, it uses tz too. Thanks guys. Andy

Worse than Y2K because of Java

This is worse than Y2K because Java needs to be patched, and JVMs proliferate on hosts like cockroaches. Older JVMs cannot be patched.

There are nearly 50 java instances on some of our hosts. The filthy little bastards hide everywhere.

Fortunately the fix can be automated and is very fast to install.

Using java's extensive built-in patch management and version management capabilities, of course.

NTP for Everyone

The NTP server system is very reliable. Its servers should also include software upgrades that clients can fetch, with an authorization system in the clients. The US NIST should produce reference standard software that the NTP servers can offer, digitally signed by NIST, and test/certify/sign 3rd party SW. And the Congress should require the insurance industry to adopt uniform standards for liability when companies don't upgrade to the industry operations standard.

This function is too important to leave to corporations that have demonstrated they upgrade themselves in their own interest only when it's a years-long campaign that everyone talks about. So it's time to automate the process. Otherwise, Americans and others in the global economy will pay much higher costs in damage and loss later, cleaning up the mess.

No big deal if you don't update...

You'll catch up to the rest of us in three weeks!

Apple

Apple just pushed an update through Software Update that fixes potential daylight saving time problems. You can grab it here [apple.com] if you use Tiger, or here [apple.com] if you still use Panther. It also released a similar update for Java. here [apple.com] is the Tiger version and here [apple.com] is the Panther one.

daylight savings time is stupid

i recognize the interest in giving people more daylight hours for harvest/ farming purposes... and how that's still nice in a service/ industrial work setting to have barbeque time after work

but why does that mean that time has to be shifted twice every year? why not just shift time by an hour once, just one time, and be done with the nonsense forever? why is it necessary to go back to "real" time in the winter?

heck, sometimes i think we should redefine 6 am as 3 am. then everyone wakes up and goes to work in the middle of the "night", and, after work in the summer, you have daylight until midnight!

we're dealing with an abstract concept here. we can do whatever we want with time. we don't have to abide by some weird need to swithc back to "real" time in the winter. just shift it once in favor of farmers/ after work barbequers and be gone with it. it's just so stupid and pointless and a waste of effort to constantly shift back and forth

Re:daylight savings time is stupid

FYI...This latest "fiddling" with DST is designed to figure out just how much energy DST actually saves (if any). By adjusting the date when DST goes into effect, and measuring the difference in energy consumption compared to previous years, we should get a good idea of whether DST is providing any energy savings and whether that savings is worth the trade-offs (there are quantifiable costs associated with DST...things like increased auto accident rates around the changeover dates).

There's also some notion that in the modern world DST actually increases the country's energy consumption. DST was originally conjured up in a world where the bulk of the energy consumption came from lighting. In that world, giving people daylight at certain hours of the day reduced their need for artificial light. In our modern world, however, things aren't that simple. For one, we have lighting that uses less energy than before. But the biggest difference is that we now have air conditioners, something that uses significantly more energy than our modern lighting. In the modern world, by ensuring that there is natural light when people get home from work, we increase the likelyhood that they will need to use their air conditioners.

So, we really have no idea whether DST actually serves its purpose anymore or if it's merely an unecessary inconvenience for any modernized country. This year's change in the time that DST goes into effect will give us a good indication of whether we can eliminate DST entirely.

Verify your Linux box is correct.

% zdump -v /etc/localtime| grep 2007 /etc/localtime Sun Mar 11 09:59:59 2007 UTC = Sun Mar 11 01:59:59 2007 PST isdst=0 gmtoff=-28800 /etc/localtime Sun Mar 11 10:00:00 2007 UTC = Sun Mar 11 03:00:00 2007 PDT isdst=1 gmtoff=-25200 /etc/localtime Sun Nov 4 08:59:59 2007 UTC = Sun Nov 4 01:59:59 2007 PDT isdst=1 gmtoff=-25200 /etc/localtime Sun Nov 4 09:00:00 2007 UTC = Sun Nov 4 01:00:00 2007 PST isdst=0 gmtoff=-28800

notice that the isdst changes from 0 to 1 on March 11. This means I have the correct zoneinfo file in my system. /etc/localtime is a symbolic link to the default timezone for your machine. (users can run their own timezone with the TZ environment variable).

% ls -l /etc/localtime
lrwxrwxrwx 1 root root 30 2006-09-24 21:50 /etc/localtime -> /usr/share/zoneinfo/US/Pacific

PS - likely the steps to check this on FreeBSD are similar. Post your experiences.

Its not just "unprepared" companies

IT at my company told us several weeks ago NOT to take the Microsoft patch as they were working on their own. They rolled it out to us this last week.

It was a complete disaster. Now my calendar entries for the affected week are mostly off by an hour (not all of them mind you) while a friend who displays dual timezones now has one less timezone in the continental US - the west coast is only two hours behind the east coast. Probably he can fix this by turning it off and back on, but it looks like we will have to rebook all meetings.

Of course, one can certainly argue that correctly implemented software would not have a problem since everything would be done internally in UTC, but clearly not all software is correctly implemented.

As for the stupid change - if they had brought us into line with Europe there would have been some logic to the change. This one was just make work for a cheap political stunt.

Windows and CMOS clock

Why does Windows not run UTC on the CMOS clock? Doing so would solve all of this "The computer has changed the clock" twice a year. The clock wouldn't be changed, just synced every now and then, but the displayed time would automatically be adjusted. POSIX and MacOS does this correctly, and 99.99% of Mac users don't even realize their CMOS clock runs UTC. Changing Daylight Time would be updating a single file, even in a closed OS like windows.

I've heard all sorts of dumb reasons against running UTC on the CMOS, like "who cares about UTC, My time is local" and "why should I keep two different times on my computer".
But, the OS will hide the UTC from you, and besides, when was the last time you used the BIOS time as your clock?

Forcing UTC on the CMOS clock is surprising since WindowsNT has used UTC for all their internal time tracking for some time. But they *calculate* it from local time, which changes twice a year, _even though_ Windows uses NTP time servers. Doh. It's gotta be *the* dumbest backward compatibility "feature". See here: http://www.cl.cam.ac.uk/~mgk25/mswish/ut-rtc.html [cam.ac.uk]

I hate to be the first to point out...

It's "Daylight Saving Time" NOT Savings...

http://en.wikipedia.org/wiki/Daylight_saving_time# name [wikipedia.org]

How to test if your linux machine is ready

> date --date="Mar 10 15:00:00 UTC 2007"
Sat Mar 10 10:00:00 EST 2007
> date --date="Mar 11 15:00:00 UTC 2007"
Sun Mar 11 11:00:00 EDT 2007

This won't set your clock or anything, it just does the timezone conversion from UTC and displays the results according to the local timezone you have selected.

Re:How to test if your linux machine is ready

> date --date="Mar 10 15:00:00 UTC 2007"
Sat Mar 10 10:00:00 EST 2007
> date --date="Mar 11 15:00:00 UTC 2007"
Sun Mar 11 11:00:00 EDT 2007

This won't set your clock or anything, it just does the timezone conversion from UTC and displays the results according to the local timezone you have selected.

Or just run zdump -v /etc/localtime | grep 2007" and make sure it says "Mar 11" instead of "Apr 1".

updated> zdump -v /etc/localtime | grep 2007
/etc/localtime Sun Mar 11 10:59:59 2007 UTC = Sun Mar 11 01:59:59 2007 AKST isdst=0 gmtoff=-32400
/etc/localtime Sun Mar 11 11:00:00 2007 UTC = Sun Mar 11 03:00:00 2007 AKDT isdst=1 gmtoff=-28800
/etc/localtime Sun Nov 4 09:59:59 2007 UTC = Sun Nov 4 01:59:59 2007 AKDT isdst=1 gmtoff=-28800
/etc/localtime Sun Nov 4 10:00:00 2007 UTC = Sun Nov 4 01:00:00 2007 AKST isdst=0 gmtoff=-32400

needswork> zdump -v /etc/localtime | grep 2007
/etc/localtime Sun Apr 1 10:59:59 2007 UTC = Sun Apr 1 01:59:59 2007 AKST isdst=0 gmtoff=-32400
/etc/localtime Sun Apr 1 11:00:00 2007 UTC = Sun Apr 1 03:00:00 2007 AKDT isdst=1 gmtoff=-28800
/etc/localtime Sun Oct 28 09:59:59 2007 UTC = Sun Oct 28 01:59:59 2007 AKDT isdst=1 gmtoff=-28800
/etc/localtime Sun Oct 28 10:00:00 2007 UTC = Sun Oct 28 01:00:00 2007 AKST isdst=0 gmtoff=-32400

Re:Linux?

Or you could put configuration data like say, time zone rules, into an external file so they could be easily updated without recompiling the kernel or rebooting. Yeah, I vote for that plan.