Monday, The Death of Websites

An anonymous reader writes "Developers implementing 'weekend inspiration' are more dangerous than hackers. Vnunet.com has this article about how eager developers and administrators create more troubles than hackers and viruses do for websites. How about those of us who start the week with a cup of coffee and the morning online-news? My inspiration and new ideas for development are definitely not the cause of the Monday-crash hour ... I think."

Great sample:)

[EnlightenedDuck]

They did a survey of 70 leading websites over a nine week period - one needs to wonder who picked those 70 leading websites, and in what sense they are considered leading or typical.

And if slashdotting causes more downtime than developer mistakes, couldn't one argue that interesting content is more harmful than bad code for website uptime?

Not just Programmers

[UCRowerG]

"However, you still get managers who don't understand the technology and want changes implemented yesterday. If it goes wrong it's the developer that ends up with egg on the face."

The article suggests that developers come back from their weekends and start fiddling with websites, but I think this last paragraph is perhaps equally or more accurate. Managers get "inspired" over the weekend just as much as code writers.

Unprofessional development

[wowbagger]

This is nothing but unprofessional development - the old "Oh, this is soooo good and sooo simple, how can it possibly cause troub..... <NO CARRIER>".

Any codebase, be it a program, a web site, or a router's firewall rules, should be changed IN TEST FIRST! Then you do your best to break it, and only after you and several others have had at it do you move it to production/HEAD/whatever (and hold your breath).

If you had a wonderful idea over the weekend, GREAT! Implement it in a test branch, try it out, and then move it to production. But if you merge it into the mainline without testing you are not acting professionally.

I will give the /. crew this: while their spelling may be atrocious, their grasp of grammer poor, and their fact and dup checking next to non-existent, they will put major changes to the codebase into Banjo first, then after they've been abused put them into the main /. site.

At least, some of the time.

Developers shouldn't be able to break stuff

[CTho9305]

In any properly managed environment, developers don't get to [i]touch[/i] the production environment. If they do, it should be read-only. All changes are made in the dev environment (developers can do what they want), put into test (developers are seriously limited), and then finally into production. Prod should be a physically separate set of servers from dev & test.

If stuff breaks on Mondays, either someone is skipping steps, or there is more going on.

Re:Not just Programmers

[AVee]

Yes and guess who get to decide what is going to be implemented and when thats going to happen?
Like:
No, Im sure it's a great idea, so just do it. I want it done before 10am.

Am I crazy?

[cruppel]

or do those developers need to possibly develop on a copy of the website not accessible to the public? I mean, it's not hat hard to hit cp -R and transfer your updated functioning website to the primary directory...Maybe I'm the only one that doesn't tinker with things that people are hitting as I type.

It's better than Friday, stupid.

[KFury]

The tone of the article talks about shoot-from-the-hip developers acting irresponsibly, on impulse. They're taking a recognized and thoughtful practice and painting it as irresponsibility.

Monday is the best time to implement changes to most sites. The irresponsible coder implements on Friday, when errors might not be caught, or fixed, until the next working day, after a full weekend of downtime, bugginess, or insecure behavior.

But that wouldn't make for an interesting story. News flash: updating code often results in bugs that need to be fixed. When do the authors suggest we roll out new versions?

Sounds about right

[Stargoat]

This really isn't surprising. The most dangerous person to a network is the person who has the administrator password.

Re:Unprofessional development

[Blaine Hilton]

The biggest problem I see is that sometimes a change is so simple it seems better to just go direct to the liver server, then all of sudden everything is down and you can't figure out what went wrong with the change.

Inspiration and discipline.. both needed

[mwillems]

Seems to me we are talking about several different things here.

First of all, presumably it is a good thing that people think, and get inspiration. Mon-Fri 9-5 is not the best time for thinking - this is the time for meeting deadlines, sitting in meetings, answering the phone, putting out fires, and so on. The only time most of us have to actually sit and think is the weekend. Personally, I think that should be encouraged.

The next step is implmenting what you have dreamt up. Obviously, most ideas fail - ask any patent officer. And obviulsly, implmenting a new idea without checking with colleagues, drawing it 0ut in a spec, getting that spec approved, then protoyping, testing, tuning is not ideal either. These procedures were invented for good reasons - not just to constrain the creative mind. This is where most developers fail - not in coming up with ideas, but in being disiplined in implmenting them. I hear "we cannot plan ahead, it does not work like that for us" all the time from my developers - this is always a misconception, and seems to me simply a combination of inexperience, laziness and inability... nothing that cannot be fixed!

Michael

What's next? Late night overtime produces bugs?

[ckessel]

Sheesh, next thing you know they'll start spouting nonsense like "burning the midnight oil leads to more bugs."

ebay motors- example

[Anonymous Coward]

http://pages.ebay.com/catindex/motorcycles.html as of 1246 pm most of the links on ebay motors section dont work! GO MONDAY!

Re:The cause of bugs

[spirality]

Mr. Gandhi has his cause and effect a little mixed up, and I think he's implying that new development shouldn't ever introduce new bugs, which is a little silly.

He does have a valid point about testing before putting code into production and being able to roll back changes. That's all pretty obvious stuff.

The mention of managers pressuring for changes, but not allowing for adequate testing time is also typical.

-Craig.

ohh come on..

[Squarewav]

It doesn't matter if the website was made on saturday , or wensday. anytime a website comes out with new code, its going to fuck up in the first few days. there is just no cheap way to test a website with a full load of users all with difrent OS's, web browsers, and connections. how many times has slashdot craped out when they update the slashcode

Re:What is it about developers?

[Enzondio]

I don't think stuff like this happens often to sysadmins or DBAs. How often do you come into work on a monday and decide to migrate to xfs because you read on slashdot over the weekend that SGI ported it to linux, and SGI is cool? Likewise, how often does an Oracle DBA decide on Monday to move some production tablespaces over to rawfs from cooked, because she read a whitepaper from Oracle on Saturday that talked about performance increases from raw filesystems?

Well, first of all those are all pretty big changes. No developer worth a damn would try to do something that massive over a weekend, by himself. Also in general I think there is more possibility of a small change causing big problems in development work than IT work, although this certainly does happen with IT, as I can attest to, having spent many a late night struggling with some sever setup or what have you.

Is it fair to say that sysadmins fix things and developers break them? What is different about a software engineer's brain than a systems engineers? Talk amongst yourselves :)

Again I think it comes back to the job of a developer, not being harder per se, but perhaps being ... more experimental. Much of what is done in the IT world has been done by many other people many other times and so one can draw from that experience. This is true in the development world as well, but I believe to a lesser degree.

Re:What is it about developers?

[teromajusa]

"Is it fair to say that sysadmins fix things and developers break them?"

Not in my experience. I've seen sysadmins break software by installing security patches, changing server passwords, changing firewall rules, restarting servers at the wrong time, swapping hardware, tinkering with network topology, failing to follow proper startup or shutdown proceedures, failing to perform necessary maintenance, etc. DBAs can cause just as much trouble tinkering with optimization, DB parameters, passwords, etc.

Thing is, anyone involved in the software process in any meaningful way can break it if they do something stupid, and in my experience, stupidity is not a trait confined to a particular profession, culture, religion, or ethnicity but is shared generally by all.

My "weekend inspirations"..

[xchino]

.. Have never defaced my site with the goatse guy nor deleted critical files. They may not work great at first (or at all) but they're in no way malicious. More dangerous than hackers or virii? I think not.

Stupid article

[Synn]

When you make changes to websites, they sometimes break. Anytime you introduce change into a stable system, you open the door for instability.

And generally business websites don't see as much traffic on the weekends, so naturally the weekend is the time to make changes.

So wow, it's no shock that Mondays are when you're most likely to see problems with a website. But these problems and hiccups are the price you pay for progress.

If you don't want to chance any disruption in your life, then I guess you should never change. Otherwise, get over it.

Need 4 environments Dev, UAT, IT & prod

[crovira]

You need to get the code monkey off the production box.

They need a Dev environment. And THAT's ALL they touch. They deliver their code to UAT.

QA needs 2 environments:
- Unit Acceptence testing (UAT) and all bugs go back to Dev
- Integration Testing (IT) and all bugs back to Dev or you need SysAdmins who need to hack the OS middleware &| environment)

Production where NOHING is allowed until its gone through UAT & IT.

Re:QA

[Hentai]

4. Managers must know the test/ QA process should never by bypassed -- this unfortunately is probably the hardest point. :-(

Of course it is. If the code breaks, it's the developer's fault, even if the Manager broke the process. The Manager can FORCE it to be the developer's fault, so what reason does the manager ever have to accept blame?

When you have a choice between waiting 3 days for a fix that has to be done TODAY, and risking breaking something that absolutely MUST NOT BE BROKEN, it is your fault, as a developer, for the manager having to make that choice at all, and either way they go, you will take the blame for the negative impact of their choice.

It's called slavery. Get used to it.

Re:I would fire the IT staff

[HeghmoH]

Yeah, that's great, your entire network has amazing reliability.

Oh, but you don't do anything interesting on it.

Have you considered that running complicated software that your business relies on could reduce reliability simply because it's actually doing something more interesting than serving internal files and transferring e-mail?

Weekend Inspiration?

[doomicon]

Is this really a case of "Weekend Inspiration", or a case of management pushing changes that haven't been thouroughly tested?

I find it quite disturbing how these companies are blaming downtime on developers. This means that:

a. You have no change control over your environment, and developers can do as they please, hence poor management.

b. Developers are implementing changes that haven't been thouroughly tested. Again poor management.

Technology and competition isn't moving so quickly that you cannot take the time to use a test/qa environment.

Re:Unprofessional development

[wowbagger]

... sometimes a change is so simple it seems better to just go direct to the liver (sic) server, then all of sudden everything is down ....

And that is EXACTLY my point. I've been bit by that too many times myself - "Oh, this is so simple I'll just roll it in". Then all goes to hell.

True, there are things that won't break until they go live - that's life in a universe ruled by Murphy. But the idea is to reduce the likelihood of an error as much as possible.

That is why, no matter how tempting, no matter how trivial the change seems, no matter how much it seems like a waste of time, you put it into test first, test as much as you can, then go live.

Have they heard of QA?

[phliar]

What kind of mickey-mouse operation is it that allows someone's (whether management or developer) mistake to take down their web site? Have they heard of QA and testing? You'd have to be insane to allow any changes to a production system without it being tested on the test system first. In all the places I've worked at (I'm a back-end hacker, using C/C++ and java) anyone who made a change to the production system without following all the test procedures (regression tests and QA signoff) would be canned in a second. (Unless it's the VP of Engineering -- but that's another story.) Or these personal sites they're talking about?

Re:Not just Programmers

[Yakko]

Wouldn't it be great if managers were actually held accountable for things like this? If they stood the chance of being fired, they may just think about the feasability of their idea before ordering it right into production...

Re:Developers shouldn't be able to break stuff

[Zalgon 26 McGee]

And they're right to say it wasn't tested thoroughly. Your test machines MUST be identical to the deployment machines; the directory structures, the version and patchlevel of the OS and critical apps...

Saying "Well, it works on MY machine" is irrelevant - the only box that counts is production.