60 Years of Hamming Codes

swandives writes "In 1950 Bell Labs researcher Richard W. Hamming made a discovery that would lay an important foundation for the modern computing and communications industries — coming up with a method for performing computing operations on a large scale without errors. Hamming wrote about how self-checking circuits help eliminate errors in telephone central offices. He speculated the 'special codes' he proposed — which became known as Hamming codes — would only need to be applied to systems requiring unattended operation for long periods or 'extremely large and tightly integrated' systems where a single failure would incapacitate the entire installation. Hamming code was the first discovery in an immense field called coding theory. This article looks back on the history of Hamming codes, their applications, and includes interviews with Todd Moon, Professor of electrical and computer engineering at Utah State University and David MacKay, Professor of natural philosophy in the department of Physics at the University of Cambridge and chief scientific adviser to the UK Department of Energy and Climate Change. An interesting read, about a little-known but fundamental element of information theory."

News For Nerds

[DarkKnightRadick]

News for nerds, stuff that matters.

This submission qualifies.

Re:News For Nerds

[starfishsystems]

Hah! It's fundamental computer science, and its use is ubiquitous. It's the equivalent of Watson and Crick's discovery of the helical structure of DNA, which we learn about in elementary school.

You want to celebrate the history of ring theory, now that would qualify as nerdly.

Re:News For Nerds

[DrSkwid]

Did they tell you Crick was high on LSD at the time [hallucinogens.com] ?

Re:

[starfishsystems]

Evidently then, highly accomplished and rational people can be in favor of hallucinogens. Not that I ever doubted it, but it's always nice to have evidence.

Re:

[starfishsystems]

P.S. Alan Turing was gay, as we know now. We all learned of his enormous contributions to computer science, but when I was an undergrad it was never mentioned that the British government crushed him anyway.

Re:

[DarkKnightRadick]

True, but it is stuff that matters. Sounds as if without Hamming Code (or whatever it would have been called had someone else discovered it) we wouldn't be where we are today.

Re:News For Nerds - Read his book!

[refactored]

His book Coding and Information Theory [amazon.com] is by far the best written and most readable hard science textbook I ever had in my university career. Read it if you want to understand the subject, read it if you want to understand how to write a good textbook!

Re:

[e70838]

Information theory was not my option, but I was in Telecom Bretagne when turbocodes [wikipedia.org] were discovered and many theoretic aspects has been demonstrated. Did the turbocodes make the Hamming code obsolete or do I miss something ?

Re:

[decipher_saint]

Sad thing is that the story has been up for nearly 2 hours and there's less than 20 comments...

Re:

[DarkKnightRadick]

I know. ):

Re:

[Anonymous Coward]

News for nerds, stuff that matters.

This submission qualifies.

And got 26 comments... while

    Ask Slashdot: Have I Lost My Gaming Mojo?

Got 300 plus...

Nerds are devaluated nowadays or Slashdot got some entirely different demographics
Maybe is time for changing the tagline?

Re:

[DarkKnightRadick]

That's what I'm thinking (both, geeks are devaluated and /.'s demographics have changed).

Re:

[syousef]

It's not news if it's decades old. The author admits in his article that the codes have well and truly been superseded. What it's clear to me is that he's trying to publicise his book (he does link to the free PDF version - so I doubt his only motivation is money). He has reviewed his own book on Amazon twice: "Brings theory to life" and "An exciting and up-to-date text"

http://www.amazon.ca/product-reviews/0521642981/ref=cm_cr_dp_all_helpful?ie=UTF8&showViewpoints=1&sortBy=bySubmissionDateDescending [amazon.ca]

Re:Little known?

[$RANDOMLUSER]

That's right. Had Hamming's discovery been more well known, he might have won the Claude E. Shannon Award [wikipedia.org].

DUCWIDT?

Re:

[wanax]

Well, of course, he did win the Richard W. Hamming Medal [wikipedia.org] 16 years later. I think the issue with Hamming is that he was so productive, and at the forefront of the field for so long, that he never settled into the "grand old man" role that tends to attract awards. I have a pet theory that for most researchers, winning big awards is a signal of their decline, because the politics of award committees means that its rare for somebody who still publishes controversial and original research to survive the nominati

Re:

[Anonymous Coward]

If you know anything about information theory, you know about hamming codes.
If you know anything about information theory, you know about hamming codes.
If you know anything about information theory, you know about hamming codes.
If you know anything about information theory, you know about hamming codes.
If you know anything about information theory, you know about hamming codes.

Re:Anonymous Coward

[Teancum]

I'm sure a whole bunch of people know about Bill Gates and Steve Jobs, but their impact upon computing and information theory is comparatively minor compared to Richard Hamming. There are many such individuals who if you have studied your history of computing that really ought to be much better known but little is really talked about even in computing circles. Usually there is a theorem or algorithm which bears the name of an individual like Dijkstra's algorithm, but who really knows much about Edsger Dijkstra, the guy who came up with the concept in the first place, or for that matter even knows the names behind the LZW compression algorithm?

If you went to a group of college seniors in computer science, how many of them would have ever heard about Grace Hooper? First classmen in the Naval Academy? (I would sure hope that the U.S. Naval Academy at least would have taught their computer science cadets something about Admiral Hooper, especially if they get assigned to the USS Hooper).

There are a bunch of people you should know in the history of computing, and unless you have a very good professor who doesn't claim to have invented the integrated circuit and every other part of computing, you generally don't know the whys for how most concepts in computer science were ever derived.

Re:

[ZERO1ZERO]

Exactly what I was thinking.

Re:Anonymous Coward

[joshki]

There is no Admiral Grace Hooper. There was, however, an Admiral Grace Hopper, who has a few things (including Hopper Hall at Dam Neck), named after her.

Re:

[Jay L]

There is no Admiral Grace Hooper

OK, but there was Mr. Hooper on Sesame Street, whose death was handled with grace, which was admirable. So that's almost the same thing.

Re:

[NoOneInParticular]

Actually, Hamming is a bit overrated. He was a tireless self-promotor who himself named these things after himself. Really a no-no. Hamming codes and Hamming distance are fairly simple constructs and by no means the first, the last, or the most significant in the field. The real giant in the field is of course Claude Shannon, who, in my opinion, has been more important for the field of computing as a whole than even Alan Turing himself (Shannon's master thesis for instance proved how to do arbitrary boolean

Re:

[CODiNE]

If you went to a group of college seniors in computer science, how many of them would have ever heard about Grace Hooper? First classmen in the Naval Academy? (I would sure hope that the U.S. Naval Academy at least would have taught their computer science cadets something about Admiral Hooper, especially if they get assigned to the USS Hooper).

A very nice post but it made me go "Ouch!" every time you called her "Hooper".

Re:

[Sulphur]

Terry Welch.

Re:Correcting Grace's name

[geekcoach]

Not to hijack the information about important work by Hamming, but those interested in Grace will have an easier time locating information about her searching on Grace Hopper, Navy Rear Admiral, Lower Half. Her 1986 David Letterman interview: http://www.youtube.com/watch?v=57bfxsiVTd4 [youtube.com]

Re:

[t14m4t]

Midshipmen majoring in Computer Science at the US Naval Academy (my major and alma mater, class of '00) are indeed cognizant of Admiral Hopper, though I don't think there's anything specifically that teaches about her contributions. Part of this (and here I start to hypothesize) is the relative age - ADM Hopper's contributions, though extremely important and noteworthy, are relatively recent, in comparison to the rest of what goes on at USNA - the goal is, after all, to provide highly technically-trained gr

See Feynman's Lectures on Computation

[radio4fan]

Feynman's excellent book 'Lectures on Computation' has a fantastic explanation of Hamming codes and distance, error correction etc.

If you're even remotely interested in information theory you *must* read this book! No prior knowledge required.

If you're a cheap bastard I'm sure you can find a pdf, but it's well worth the asking price.

Re:See Feynman's Lectures on Computation

[puto]

Anything by Feynman is worth it. I am 41 years old and i remember seeing super 8 videos of him in grammar school from my science teacher. Brilliant and engaging.

Re:See Feynman's Lectures on Computation

[Rockoon]

For your enjoyment

Feynman - The Douglas Robb Memorial Lectures [vega.org.uk]

Anything by Feynman...

[refactored]

... is excessively vague and handwaving requiring literally hours and pages of close work to fill in the gaps between the equation N he shows and the next equation N+1 that he says follows from N.

Yup, a brilliant guy I'm sure, but not the guy I want teaching me. At the end of a course, (call me greedy), _I_ want to know how to do everything in the course, not merely have a warm fuzzy WEE-WOW feeling that something exciting just went by that I can't quite reproduce.

Give me Richard Hamming's books instead of Feynman's any day. Ok, they won't make you go "WEEE WOW!!!", but on the other hand you will have an excellent understanding of the material AND be able to reproduce and USE any result in them yourself.

Re:

[godunc]

At the end of a course, (call me greedy), _I_ want to know how to do everything in the course, not merely have a warm fuzzy WEE-WOW feeling that something exciting just went by that I can't quite reproduce.

Holy flamebait; you are talking about America's most famous physics teacher. You were expecting a "how to" book from a nobel laureate?

Re:

[refactored]

Flame bait nothing, wake up call to see past the hype... yes.

Maybe if I was a Nobel Laureate material... I'd agree with you.

However, Feynman is not a suitable teacher neither for I, nor for that 99.9% segment of Humanity we call "mere ordinary mortals".

My wife, bless her, is quite capable of digesting and following Feynman's books, but then her skills are "world class" (to me "goddess-like").

By far most maths / physics university graduates, let's be honest now, get a "Whee! Wow! That was exciting!"

Re:

[travisco_nabisco]

I'll have to look this up. I took an undergrad engineering elective in Error Coding and found it to be one of the most fascinating subjects I have been exposed to. The mathematics behind it really are amazing.

Re:See Feynman's Lectures on Computation

[martin-boundary]

If you're a cheap bastard I'm sure you can find a pdf, but it's well worth the asking price.

Exactly. Remember kids, the money you spend goes directly to Richard Feynman, so he can continue to write excellent books from beyond the grave.

TFA mentions Mackay's book.

[serviscope_minor]

TFA mentions Mackay's book. It is an awesome book, and is free online. I have a dead-tree copy, too. Well worth the price.

Re:

[rrohbeck]

I just downloaded it.
http://www.inference.phy.cam.ac.uk/mackay/itila/book.html [cam.ac.uk]

Another Mackay book.

[northerner]

Mackay also has another book which may be interesting.

"Sustainable energy - without the hot air", available as a free PDF download.

I haven't read it yet, but I will given his credibility with the article and other book.

http://www.withouthotair.com/ [withouthotair.com] or http://www.inference.phy.cam.ac.uk/withouthotair/ [cam.ac.uk]

Here is a podcast of a lecture he gave at Cambridge on the topic of sustainable energy.

http://mediaplayer.group.cam.ac.uk/component/option,com_mediadb/task,play/idstr,CU-CSF-Lectures_2008-12_David_MacKay/v [cam.ac.uk]

Re:

[$RANDOMLUSER]

Not to burst your delusions, but MUCH of the Bell Labs research in this area had to do with communications (i.e. telephone) SATELLITES.

first thought.....

[hoytak]

Hamcode, hamcode, where you been? Around the world and I'm going again...

Not the First Discovery in Coding Theory

[asnelt]

Hamming code was the first discovery in an immense field called coding theory

First discovery? I would say Shannon's historic paper on coding theory "A Mathematical Theory of Communication" from 1948 was earlier.

Re:

[scatter_gather]

To summarize the article that you seemed not to have read, Shannon is cited as writing the seminal paper to which you refer, and in it created an existence proof for error correction codes. He did not, in his paper, actually go so far as to create an ECC. According to TFA, Shannon is credited with creating the entire field of information theory. Not a bad accomplishment. Hamming was noted as actually creating ECCs and laying the foundation stone for coding theory. It's probably why they named the codes af

Re:Not the First Discovery in Coding Theory

[rrohbeck]

Shannon only proved that those codes exist. Hamming gave the first examples. So it's fair to say that Shannon was about information theory, not coding theory.

Re:

[epine]

Shannon's work was general over the error model. Coding theory assumes a specific error model (such as bit error rates, insertion, deletion, magnitude distortion).

It doesn't take much wits to translate Shannon's work into a rudimentary code. Constructing codes near the bounds of optimality is extremely difficult, especially if the decoder corrects errors with better efficiency than combinatorial trial and error.

I wouldn't say Shannon's work was light on coding theory, much of which was implied pretty dire

Re:

[serviscope_minor]

By the time you're correcting 5 bit errors in a 256 bit packet, you don't want to have to search for the nearest valid codeword combinatorially.

The best codes, LDPCC, can only be decoded optimally using combinatorial search, the question "is this the optimal decode?" being NP-complete. So, they always use heuristic decoders, generally loopy BP. Fortunately, that is extraordinarily suitable for efficient hardware implementation. But it gives no bounds on error rate, nor even a guarantee of convergence.

At thi

More info on Dr. Moon

[aquila.solo]

For the record, Dr. Moon is the Department Head of ECE. He gives killer lectures, but his tests will make you wish for death ;-).

I'm currently working on an MS there. Good to see him get some publicity.

Easy to understand

[elbow_spur]

Hamming codes are practical things, while Shannon's analyses of codes were more abstract (though still hugely useful and important)
Consider the checksum bit. It helps to catch errors but there are 2 problems. First, if there is a double error (more likely if the checksum is on a longer string), then the error isn't caught Second, even if we know there is an error, we can't recover, but have to resend.
The easiest error-correcting code is to replace every bit with a triple copy of itself. So
101 become

Re:

[elashish14]

Not to mention that it's a very small footprint: for N bits in a transmitted message, you only need log(N,2) parity bits to retain the same error correction/detection capability. You can pretty easily balance how robustly you want to protect your data with how much excess information you want to transmit.

How to do research like Hamming

[knutert]

Want to be like Hamming? Here's how:
In summary, I claim that some of the reasons why so many people who have greatness within their grasp don't succeed are:
        * they don't work on important problems,
        * they don't become emotionally involved,
        * they don't try and change what is difficult to some other situation which is easily done but is still important,
        * and they keep giving themselves alibis why they don't.
        * They keep saying that it is a matter of luck.
I've told you how easy it is; furthermore I've told you how to reform. Therefore, go forth and become great scientists!

Source: http://paulgraham.com/hamming.html [paulgraham.com]

Re:

[sahai]

http://alum.sharif.edu/~mynaderi/Claude%20Shannon.html [sharif.edu]

That is an online transcription of Claude Shannon's thoughts on the matter. Google pointed to this link from Sharif, but you can see a printed version in the second volume of his collected works, I believe. I ran across this during my blissful graduate student days.

I love how this talk ends with him asking his audience to come and look at this machine that he built.

Reflections on Hopper and Hamming

[perry64]

In my Naval career, I was lucky enough to come across both of these titans of computing’s early age. RADM Hopper gave a lecture to every plebe class at the Naval Academy , including mine in 1984, where she would give each Midshipman a short length of wire of the length that light traveled in a nanosecond. She used these to illustrate stories she told of the early days of computers that were programmed by connecting wires differently. Her speech was the first place I heard “it was easier to b