Slashdot Log In
Psychologist Beating Math Nerds in Race to Netflix Prize
Journal written by s1d (1185389) and posted by
Zonk
on Tue Mar 04, 2008 11:19 PM
from the must-calculate-harder dept.
from the must-calculate-harder dept.
s1d writes "An almost-anonymous British psychologist named Gavin Potter has suddenly risen to the top of the Netflix prize charts. With his very first attempt, he got a score which took the BellKor team seven months to reach. Currently at a score of 8.07, he has only five teams ahead of him now in the race for the ultimate Netflix algorithm. 'Potter says his anonymity is mostly accidental. He started that way and didn't come out into the open until after Wired found him. "I guess I didn't think it was worth putting up a link until I had got somewhere," he says, adding that he'd been seriously posting under the name of his venture capital and consulting firm, Mathematical Capital, for two months before launching "Just a guy." When he started competing, he posted to his blog: "Decided to take the Netflix Prize seriously. Looks kind of fun. Not sure where I will get to as I am not an academic or a mathematician. However, being an unemployed psychologist I do have a bit of time."'"
Related Stories
[+]
News: Build a Better Netflix, Win a Million Dollars? 197 comments
An anonymous reader writes "In a quest to better movie recommendations, Netflix is opening their database (nytimes, registration and first child required) to users to try to craft a better recommendation technology. The problem is not easy. Says one researcher: 'You're competing with 15 years of really smart people banging away at the problem.'" Recommender systems are really an interesting problem, and that is likely very interesting data to play with.
[+]
Netflix Prize Competitor Already Beats Netflix 174 comments
Baldrson writes "Within the first week of the announcement of The Netflix Prize a team has already beaten Netflix's own movie recommendation algorithm. This is pretty impressive given the previously quoted researcher who said: 'You're competing with 15 years of really smart people banging away at the problem.' The team is WXYZConsulting.com apparently registered by a data mining professor named Yi Zhang. Congratulations are in order for Netflix and Prof. Zhang's team who are demonstrating, yet again, the power of prizes to accelerate progress."
[+]
Close but no Cigar for Netflix Recommender System 114 comments
Ponca City, We Love You writes "In October 2006, Netflix, the online movie rental service, announced that it would award $1 million to the first team to improve the accuracy of Netflix's movie recommendations by 10% based on personal preferences. Each contestant was given a set of data from which three million predictions were made about how certain users rated certain movies and Netflix compared that list with the actual ratings and generated a score for each team. More than 27,000 contestants from 161 countries submitted their entries and some got close, but not close enough. Today Netflix announced that it is awarding an annual progress prize of $50,000 to a group of researchers at AT&T Labs, who improved the current recommendation system by 8.43 percent but the $1 million grand prize is still up for grabs and a $50,000 progress prize will be awarded every year until the 10 percent goal is met. As part of the rules of the competition, the team was required to disclose their solution publicly. (pdf)"
[+]
IT: Anonymity of Netflix Prize Dataset Broken 164 comments
KentuckyFC writes "The anonymity of the Netflix Prize dataset has been broken by a pair of computer scientists from the University of Texas, according to a report from the physics arXivblog. It turns out that an individual's set of ratings and the dates on which they were made are pretty unique, particularly if the ratings involve films outside the most popular 100 movies. So it's straightforward to find a match by comparing the anonymized data against publicly available ratings on the Internet Movie Database (IMDb) (abstract on the physics arxiv). The researchers used this method to find how individuals on the IMDb privately rated films on Netflix, in the process possibly working out their political affiliation, sexual preferences and a number of other personal details"
[+]
Science: Interest Still High In the Netflix Algorithm Competition 77 comments
circletimessquare brings us an update to the status of the million-dollar Netflix competition to develop a better algorithm for movie recommendations. We've discussed aspects of the competition since it started two years ago, but the New York Times has a lengthy overview of where it stands now.
"The Netflix competition is still going strong, with a vibrant, competitive roster of some 30,000 programmers around the globe hard at work trying to win the prize. The Times provides a look at some of the more obsessive searchers, such as Len Bertoni, a semi-retired computer scientist near Pittsburgh who logs 20 hours a week on the problem, oftentimes with the help of his children. There's also Martin Chabbert in Montreal: 'After the kids are asleep and I've packed the lunches for school, I come down at 9 in the evening and work until 11 or 12.' The article gets into the history of the search algorithm Netflix currently uses, and explores the hot commodity called 'singular value decomposition' that serves as the basis for most of the algorithms in competition."
This discussion has been archived.
No new comments can be posted.
The Fine Print: The following comments are owned by whoever posted them. We are not responsible for them in any way.
Full
Abbreviated
Hidden
Loading... please wait.
Beating everyone? (Score:5, Funny)
Re:Beating everyone? (Score:5, Funny)
Welcome to your therapy session.
Surely not everyone?
Can you elaborate on that?
Why do you mention computers?
Why do you mention computers?
Does that question interest you?
Parent
Domain Knowledge (Score:5, Insightful)
Re:Domain Knowledge (Score:5, Funny)
Like they really had a choice...
Parent
What's the point? (Score:5, Interesting)
And yes, I have karma to burn. Yes I do.
Re:What's the point? (Score:5, Funny)
Parent
Average psychologist (Score:5, Informative)
that had an undergraduate degree in psychology, a masters degree in operations research [wikipedia.org] that after being well employed for a number of years -- "In 2006, he left his job at IBM to explore the idea of starting a PhD in machine learning, a field in which he has no formal training. When he read about the Netflix Prize, he decided to give it a shot -- what better way to find out just how serious about the topic he really was?"
Misleading summary (Score:5, Informative)
Interdiscplinary approach (Score:5, Interesting)
Re:Interdiscplinary approach (Score:4, Interesting)
Parent
Re:Interdiscplinary approach (Score:5, Interesting)
I work for a small company. My current job isn't stable, and doesn't pay well, so I'm taking an IT course so I can land a (hopefully) stable job/career.
However... in my current job I wear all kinds of hats. Server's down? I'll fix it. Marketing materials need to be designed? I'll do it. Proposal needs to be edited? I'm there. Computer needs more RAM? I'll install it. Photo of product needs to be masked-out? Done. Need to do some research? I'll get on it.
The kind of job I'm being trained for... I'll be stuck on the straight and narrow, handling one sort of task. When companies want an IT guy, they want an IT guy. I don't know how I'll be able to handle that. I LIKE having different responsibilities. I don't want to be one guy on an org chart with a specified duty.
Blah. I really went on a tangent there...
Parent
Re:Interdiscplinary approach (Score:5, Insightful)
If you aren't careful, you end up being the fall guy for a widening array of mishaps.
For instance, you help set up the video projector a couple of times for presentations. Then during the next presentation the projector fails. In some eyes it will be your fault, because you're now the "PowerPoint Guy." Nevermind that the bulb was past its recommended use hours, or that the presenter forgot his VGA dongle, or whatever.
It seems like if I want to come out as the go-to guy for some area of tech, I'd damn well better get up to pro-level speed really quick. Because soon I'm going to have to be mitigating crises and solving complex problems that before were just chalked up to "well, that thing's always been a problem." Yeah, now it is *your* problem!
Parent
WWLD? (What would leonardo do?) (Score:5, Funny)
Oh, I don't know. If only we had a art history & statistics dual major to figure it out...
Parent
Re:Uh, no... (Score:5, Insightful)
Furthermore, because early scientists did not have as much to build on, that makes it all the more difficult. Where was Faraday to get his inspiration on lines of force? What lead Maxwell in the right direction to unifying light with electromagnetism?
It's great that 3rd graders know about electric circuits. That's the point of scientific progress. That doesn't make the original task trivial in any sense.
In other words, I hate you.
Parent
All of 18th century science? Really? (Score:5, Insightful)
If you know how to do all those things, good for you. My point is that a secondary school education in "science" does not by itself provide one with the same understanding as the great scientists who first among all others figured out how to do each of these things.
Parent
Umm.... (Score:5, Informative)
Psychologist? (Score:5, Insightful)
Re:Psychologist? (Score:5, Funny)
Parent
Re:Psychologist? (Score:5, Funny)
Parent
Re:Psychologist? (Score:5, Funny)
Parent
Re:Psychologist? (Score:5, Insightful)
Parent
Thinking about how people shop (Score:5, Interesting)
So, if I was to setup a movie viewing for them. I'd setup something along the lines of a Genre, rating(R,PG-13,G), Alphabet.
It's kind of a takeoff on my video game organization method that increases sales of video games by 30 percent. I called it ABSRG short for Alphabetize By Section(4 foot section), Rating(M on top T in the middle and E towards the bottom.), Genre(Sports, driving, shoot em up). Please note, this cannot be patented, I already let it go out for more than a year(Started in 1998)and I have the pictures and time notes to prove it.
Free Idea (Score:5, Interesting)
Okay, I'm not trying for this prize, but there's one thing about Netflix "recommendations" that bugs me so I'm throwing out this complete freebie of an idea. If it helps someone get a 0.001% improvement to add this ONE little additional check, great.
I am learning Japanese. I have been watching several hundred Japanese-language movies for the past couple years. I don't watch movies in Greek, Spanish, Turkish, Farsi, Italian, Russian, German, or Hebrew. I did watch Amelie four years ago but that doesn't mean I love French movies. Most of my recommendations are for foreign films, but only a small fraction of those recommendations are for Japanese movies.
Apparently, Netflix doesn't have a column in their database saying WHAT language a movie uses principally, it just has a flag saying it is not English. It's the only explanation I can see for not checking for such a strong correlation. I admit, I might not be sharing the experience of the most common movie-renting drone in the bunch, but I doubt I'm the only person who has such a lopsided taste in movies. If the language (or alternate soundtrack languages) ARE known in the database, please see if the renter has a bias for movies in a particular language.
Almost anonymous? (Score:5, Funny)
That's true (Score:5, Interesting)
I've also found this to be true. Lol, I actually knew people in college that did nothing but program computers in their spare time, and took psych because it was easy, wouldn't distract them and gave them more time to do the programming they wanted. They didn't ever expect to practice psych.
Parent