Psychologist Beating Math Nerds in Race to Netflix Prize

s1d writes "An almost-anonymous British psychologist named Gavin Potter has suddenly risen to the top of the Netflix prize charts. With his very first attempt, he got a score which took the BellKor team seven months to reach. Currently at a score of 8.07, he has only five teams ahead of him now in the race for the ultimate Netflix algorithm. 'Potter says his anonymity is mostly accidental. He started that way and didn't come out into the open until after Wired found him. "I guess I didn't think it was worth putting up a link until I had got somewhere," he says, adding that he'd been seriously posting under the name of his venture capital and consulting firm, Mathematical Capital, for two months before launching "Just a guy." When he started competing, he posted to his blog: "Decided to take the Netflix Prize seriously. Looks kind of fun. Not sure where I will get to as I am not an academic or a mathematician. However, being an unemployed psychologist I do have a bit of time."'"

What's the point?

[causality]

Is there some merit to this story other than "your sterotypes can be wrong", which is itself cliche enough to be considered a stereotype in its own right? I like Henry David Thoreau's explanation of such trivia:

And I am sure that I never read any memorable news in a newspaper. If we read of one man robbed, or murdered, or killed by accident, or one house burned, or one vessel wrecked, or one steamboat blown up, or one cow run over on the Western Railroad, or one mad dog killed, or one lot of grasshoppers in the winter -- we never need read of another. One is enough. If you are acquainted with the principle, what do you care for a myriad instances and applications? To a philosopher all news, as it is called, is gossip, and they who edit and read it are old women over their tea. Yet not a few are greedy after this gossip. There was such a rush, as I hear, the other day at one of the offices to learn the foreign news by the last arrival, that several large squares of plate glass belonging to the establishment were broken by the pressure -- news which I seriously think a ready wit might write a twelve-month, or twelve years, beforehand with sufficient accuracy.

And yes, I have karma to burn. Yes I do.

Interdiscplinary approach

[oceaniv]

The topic is incredibly fascinating. And just a thought, up to 5 centuries ago "scientists" were incredibly versatile people, with mastery over a few fields at a time... A lot of people argue that this was out of necessity, but could the versatility have been important development of multiple renaissances (In Greece, East/West Asia, and Europe)? And could the bottleneck specialization of fields that has occurred in the past three centuries simply be period of transition/stifling new ways of thinking? Could the emergence interdisciplinary experts lead to another 'renaissance' of sorts?

Thinking about how people shop

[GregPK]

My extensive retail experience says people like to shop by the following methods. Ratings, Genre, Alphabet.

So, if I was to setup a movie viewing for them. I'd setup something along the lines of a Genre, rating(R,PG-13,G), Alphabet.

It's kind of a takeoff on my video game organization method that increases sales of video games by 30 percent. I called it ABSRG short for Alphabetize By Section(4 foot section), Rating(M on top T in the middle and E towards the bottom.), Genre(Sports, driving, shoot em up). Please note, this cannot be patented, I already let it go out for more than a year(Started in 1998)and I have the pictures and time notes to prove it.

That's true

[Crazy Taco]

Very few psych majors do psych professional. I have a BA from U. of Houston, which I have used to do tech support and sales for a software company, and developed my own FPS as a solo project.

I've also found this to be true. Lol, I actually knew people in college that did nothing but program computers in their spare time, and took psych because it was easy, wouldn't distract them and gave them more time to do the programming they wanted. They didn't ever expect to practice psych.

Re:Interdiscplinary approach

[Metasquares]

I think so [projectpolymath.org].

Re:Interdiscplinary approach

[Kenshin]

See, I hate this current mode of profession.

I work for a small company. My current job isn't stable, and doesn't pay well, so I'm taking an IT course so I can land a (hopefully) stable job/career.

However... in my current job I wear all kinds of hats. Server's down? I'll fix it. Marketing materials need to be designed? I'll do it. Proposal needs to be edited? I'm there. Computer needs more RAM? I'll install it. Photo of product needs to be masked-out? Done. Need to do some research? I'll get on it.

The kind of job I'm being trained for... I'll be stuck on the straight and narrow, handling one sort of task. When companies want an IT guy, they want an IT guy. I don't know how I'll be able to handle that. I LIKE having different responsibilities. I don't want to be one guy on an org chart with a specified duty.

Blah. I really went on a tangent there...

Free Idea

[Speare]

Okay, I'm not trying for this prize, but there's one thing about Netflix "recommendations" that bugs me so I'm throwing out this complete freebie of an idea. If it helps someone get a 0.001% improvement to add this ONE little additional check, great.

I am learning Japanese. I have been watching several hundred Japanese-language movies for the past couple years. I don't watch movies in Greek, Spanish, Turkish, Farsi, Italian, Russian, German, or Hebrew. I did watch Amelie four years ago but that doesn't mean I love French movies. Most of my recommendations are for foreign films, but only a small fraction of those recommendations are for Japanese movies.

Apparently, Netflix doesn't have a column in their database saying WHAT language a movie uses principally, it just has a flag saying it is not English. It's the only explanation I can see for not checking for such a strong correlation. I admit, I might not be sharing the experience of the most common movie-renting drone in the bunch, but I doubt I'm the only person who has such a lopsided taste in movies. If the language (or alternate soundtrack languages) ARE known in the database, please see if the renter has a bias for movies in a particular language.

Re:Thinking about how people shop

[WK2]

Maybe they can't win a court case, but they could probably get at least 1 mega$ settlement by suing Walmart, Circuit City, Fry's, Target, and Gamestop.

Re:Domain Knowledge

[Anonymous Coward]

Yes, they did. Every man has a choice between his work and his social life. Many men, great men, choose to dedicate themselves to their work. For the vast majority of these men we call them "geeks". For a smaller number, who end up making the mistake of getting married, we call them bastards. Einstein was one such bastard. He treated his wife and children terribly - and not just because he was basically a German man - but because his work came first.

Besides which, sex as described by the kind of people who think geeks really should be trying harder to get it is basically sport. Geeks have enough common sense to recognize that sport is no fun.

And?

[Secret Rabbit]

There a joke that I heard (that's actually pretty much true):

A Physicist goes to a Mathematician for advise on solving a Differential Equation. The Physicist explains this and writes the equation on a Blackboard. The Mathematician stares at the equation for more than half an hour. Finally, he says, "Yes, it has a solution."

Basically, the Maths (even applied) are about details and considering them *very* carefully. With this in mind, is it any surprise that they are somewhat "slow"? Especially when they are starting from scratch within the problem domain?

Re:Domain Knowledge

[Yold]

Exactly what I thought, even if he is beating math-nerds, you cannot create a machine-learning algorithm without using fairly sophisticated mathematics. Since psychology is largerly based on statistics, I am sure the guy has a firm grounding in the subject. I am sure he isn't basing his algorithm on Aedipus Complexes (sic) and ink blots.

Re:Domain Knowledge

[Anonymous Coward]

Aaah, but many psychologists would argue that dedication to work is simply a means to an end. Basically you become important, rise to the top if you will. And then pull a Bill Clinton. The question is not "why would a great man be adulterous and risk losing his position." in the eyes of many behavioral researchers but "Why not? That was the point of gaining the position in the first place." Note: I don't personally ascribe to this viewpoint but it's hard to argue with if you believe that there is no absolute moral code which then implies a creator/supreme engineer of some kind.

Re:Domain Knowledge

[g00nsquad]

Anything is arguable by psychologists. However, if you don't need to believe in an absolute moral code, you also don't need to believe in an archetypical behaviour pattern to which we all adhere (read: behaviour pattern that dictates everything we do is part of some elaborate mating ritual). As such, it may be the case that there are many different reasons why people dedicate themselves to work.

In the case of great scientists, artists, politicians or inventors, it may simply be about pure fascination with their particular interest (eg. Darwin studying and classifying barnacles for 8 years - 1856 to 1864), an altruistic (read instinctive if you will) desire to make living conditions better for their counterparts (eg. Norman Borlaug, 1970 Nobel Peace Prize Laureate, for his research in wheat agriculture leading to the Green Revolution), or some other reason.

In short, perhaps not everything is about f-cking, however f-cking certainly is everything to some.

On the back of giants

[bussdriver]

Subjects get so complex and so specialized there HAS to be limits to how many giants a human can climb and still have time enough to become a giant themselves. Abstraction helps to greatly extend this range; the people behind abstractions/simplifications may not be considered giants because they do not produce progress themselves but just facilitate others so they can extend their reach into the unknown.

There does not appear to be that many 'giant' scientific figures anymore despite the exponential scientific growth. Maybe it is just an appearance and there are more; but are there more proportionally to the number scientists?

How many big leaps in knowledge have been made in the old fields like physics for example? If the decline does not exist now, won't it exist at some point??

Re:Interdiscplinary approach

[Antique Geekmeister]

Or could the existence of things called "printed books" lead to the spread of knowledge among others who could use it, rather than having to build up their theoretical knowledge from personal experience and a few precious manuscripts? And make it more broadly available so that people with less than mutant brilliance can contribute andn publish their contributions for others?

I don't think you need to read in any great cultural change to a cross-disciplinary approach here: the problem is one well-suited to this man's exact skills, an algorithmic computation of likely human behavior. Encouraging cross-disciplinary work because people focused too tightly on one field will miss available tools from other fields is helpful, and certainly helps keep me paid for work that ranges throughout computing and engineering fields.

Re:Domain Knowledge

[droptone]

You'd be surprised how many people in psychology are math-phobic and really only know the basic stuff they need to do research (z tests, t tests, ANOVA, regression, power for grant requests, etc) without any deeper understanding of the underlying concepts. If the guy went through a decent quantitative program then he'd have a much better understanding of math since they usually prefer the applicants to have several levels of calculus plus a handful of actual statistics courses (and not stat for psyc), along with the stuff he learned there.